Hi all, I am planning to deploy the data hub with ...
# getting-startedc
calm-student-43079
10/13/2022, 5:21 PMHi all, I am planning to deploy the data hub with integrates to HUDI and delta on Azure ADLS Gen2 are there any quick start documentation which can help me for the same?
m
mammoth-bear-12532
10/15/2022, 3:33 PM
Hi @calm-student-43079 are you looking for integration guides to connect Datahub with hudi and delta lake. Or looking to deploy DataHub on azure?
c
calm-student-43079
10/15/2022, 3:47 PMHi @mammoth-bear-12532 I m looking for deploying data hub on aks (k8s) and then connect to hudi and delta lake which resides on adls gen 2
m
mammoth-bear-12532
10/15/2022, 7:32 PM
We don’t have specific guides for deploying DataHub on Azure. But you should be able to get started with the k8s artifacts that we publish (like the helm charts)
mammoth-bear-12532
10/15/2022, 7:33 PM
Wrt hudi : the hudi project hosts the docs for the integration: https://hudi.apache.org/docs/syncing_datahub/
c
calm-student-43079
10/15/2022, 7:34 PMThank you for the response
m
mammoth-bear-12532
10/15/2022, 7:36 PM
For delta-lake integration, we don’t have adls support yet (the current connector works with s3)
mammoth-bear-12532
10/15/2022, 7:36 PM
But there is quite a bit of interest in that, so we will probably work on it shortly
mammoth-bear-12532
10/15/2022, 7:38 PM
You can get around the limitation if you connect to hive directly (if you have a metastore on top of delta lake)
c
calm-student-43079
10/15/2022, 7:40 PMSo even if I have hudi in adls gen2 I can if I have hive meta store
calm-student-43079
10/15/2022, 7:40 PMAnd the same can work with delta too
m
mammoth-bear-12532
10/15/2022, 7:41 PM
Yes, the integration with hudi is a push based integration which means you get metadata pushed to Datahub whenever it changes which is nice
mammoth-bear-12532
10/15/2022, 7:42 PM
Hive integration is pull-based, which means you are extracting metadata in batch manner which makes it less fresh
mammoth-bear-12532
10/15/2022, 7:43 PM
So you can mix and match : use hudi native integration for the hudi case and hive on delta for delta tables on adls
mammoth-bear-12532
10/15/2022, 7:43 PM
Or wait a month for delta on adls to be available natively
c
calm-student-43079
10/15/2022, 7:44 PMIs there any active PR implementing the adls gen2 support ?
calm-student-43079
10/15/2022, 7:44 PMWould love to contribute if possible
m
mammoth-bear-12532
10/15/2022, 8:07 PM
That would be great
mammoth-bear-12532
10/15/2022, 8:07 PM
There isn’t an active PR yet
mammoth-bear-12532
10/15/2022, 8:07 PM
There is an iceberg source that works on top of adls today
mammoth-bear-12532
10/15/2022, 8:08 PM
So it would be ideal to refactor that out and use it across iceberg and delta sources.
c
calm-student-43079
10/15/2022, 9:19 PMPerfect thanks I would love to contribute will go through process of sending PR
b
brave-businessperson-3969
02/20/2023, 11:51 PM@calm-student-43079: Did you worked on this any further? adls gen2 support is also very interesting for us.
c
calm-student-43079
03/24/2023, 2:03 PMSorry for the late reply 😞 I am planning to work on it soon
c
crooked-airline-45849
04/04/2023, 3:15 PMHi, there!
Let me know if you guys need more help on this, I have interest in adls gen2 support too.
3 Views