Hi
QQ: Have you successfully connected to and inge...
# generala
Amir Jalali
10/31/2024, 1:03 PMHi
QQ: Have you successfully connected to and ingested data from a Delta Lake (Delta Table) on AWS or Azure?
a
Abhishek Balaji Radhakrishnan
10/31/2024, 3:29 PMFrom AWS, yes. Are you running into any issues?
a
Amir Jalali
10/31/2024, 3:29 PMim looking for azure and using abfs but its not working, have used azureStorage as well but does not work
Amir Jalali
10/31/2024, 3:31 PMjust randomly checking different “Delta table path” for azure none has worked so far
a
Abhishek Balaji Radhakrishnan
10/31/2024, 3:34 PMcan you paste the error log here? you might have to add the hadoop-adls dependency to the extension to read from azure
a
Amir Jalali
10/31/2024, 3:37 PMFailed to sample data: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem not found
Amir Jalali
10/31/2024, 3:38 PMwhat is hadoop-adls?
Amir Jalali
10/31/2024, 3:39 PMim going to download all hadoop azure libraries in hadoop dependencies folder
a
Abhishek Balaji Radhakrishnan
10/31/2024, 3:42 PMin this case, you may want this: https://hadoop.apache.org/docs/stable/hadoop-azure/abfs.html
Abhishek Balaji Radhakrishnan
10/31/2024, 3:45 PMmy suggestion would be to add hadoop-azure as a maven dependency to the extension’s pom.xml https://github.com/apache/druid/blob/master/extensions-contrib/druid-deltalake-extensions/pom.xml so it pulls the required transitive dependencies too
a
Amir Jalali
10/31/2024, 3:50 PMhmmm makes sense let me give it a try although im not really sure if that’s going to solve the issue and still not sure why azure is not included by default
Amir Jalali
10/31/2024, 4:09 PMadded as a proposal here:
https://github.com/apache/druid/issues/17441
a
Abhishek Balaji Radhakrishnan
10/31/2024, 5:03 PMThanks, let us know if you had the chance to try that out by building the extension with those dependencies? It would be good to make a pr if that worked