How to use datalake/container/buckets names and folders with Iceberg source? I think I'm at the point where I could use your help to figure out what to do with datalake and folder names... I can certainly speak for my setup and use-case, but I don't think everyone is using Iceberg the same way. I attended (got distracted a few times by work though) yesterday's townhall and I'll need to watch the video again, but I think what was presented (platform instance, container, etc...) will help. I am using Azure Datalake Gen2 with hierarchical namespaces. Here is how Azure is organized: • Datalake_A ◦ Container_X ▪︎ Folder_1 • Iceberg_Table_1 ▪︎ Iceberg_Table_2 ◦ Container_Y ▪︎ Folder_3 ◦ Container_Z ▪︎ Folder_4 • Datalake_B ◦ Container_X ▪︎ Folder_1 • Iceberg_Table_1 So you can have multiple datalakes (or storage accounts) and each datalake can have 1 or multiple containers. Each container can be organized with folders. You can see a container as a root-level folder. It is technically different, but for simplicity, it can be abstracted. Just like databases, multiple datalakes could have table name collisions just like I showed in my example. In this case, there would be 2 Iceberg tables with the same "name":

Folder_1.Iceberg_Table_2

But they would have two different Azure URLs (

abfss://{container_name}@{account_name}.<http://dfs.core.windows.net/{folder}|dfs.core.windows.net/{folder}>

): •

<abfss://Container_X@Datalake_A.dfs.core.windows.net/Folder_1/Iceberg_Table_1>

•

<abfss://Container_X@Datalake_B.dfs.core.windows.net/Folder_1/Iceberg_Table_1>

My question is how should the Iceberg source deal with this? How does it compare to AWS S3? How would it look for someone using a local filesystem?

Hi @modern-monitor-81461! I’ll make sure someone from the team takes a look at this ASAP

Yeah this makes sense to me

The datalake name (or Storage Account name) needs to be unique across all Azure tenants. The reason (or at least one of the reason) is that a storage account (

storagesample

) is accessed via a set of URLs where the domain is a Microsoft domain (ex:

<https://storagesample.blob.core.windows.net>

,

<https://storagesample.file.core.windows.net>

, etc...)

Hey @modern-monitor-81461! Checking in here - did you get the direction/input you need to move forward?

Thanks @little-megabyte-1074. I'm not sure. I guess Friday's release will help clear a few things once I get my hands on the new container feature. For me, the code I have right now is doing the job. It's not perfect and it can be improved, but it works. I don't know how far I need to drive this before I can simply hand over the keys to you if you want to add it to DataHub. We are planning to put it to test in our environment with real users in a month or so. I guess the main direction/input I'm looking for is what is missing to make it fit nicely with the other sources DataHub offers. Supporting containers is probably one of them...

Awesome, ok - we’ll definitely be including Containers in the next release; John just merged it into Master if you want to start playing around with it beforehand https://github.com/linkedin/datahub/pull/4037 https://github.com/linkedin/datahub/pull/4019