we are implementing a S3 lakehouse on athena/spark...
# integrate-iceberg-datahubw
wide-optician-47025
03/21/2023, 4:05 PMwe are implementing a S3 lakehouse on athena/spark with iceberg tables; I would like to be able to ingest the iceberg tables
m
modern-monitor-81461
03/21/2023, 4:16 PMHi @wide-optician-47025 the current codebase only supports Azure datalake. This limitation was originally imposed by the legacy Python Iceberg implementation. I have been holding on a new code update that leverages the new pyiceberg library, which will support AWS and ADLS as well as different Catalog implementations. The current code base only support HadoopCatalog.
I am waiting on release 0.4.0 of pyiceberg to submit my new PR to DataHub. I did not test my ingestor with any AWS based infra since I do not have such access, but pyiceberg was developed for AWS first, so it should work.
w
wide-optician-47025
03/21/2023, 4:25 PMhello, thanks for the update
wide-optician-47025
03/21/2023, 4:26 PMwe are in the design phase of the project, so just created some test iceberg tables ; I am excited to be get the S3 iceberg ingestion capability
wide-optician-47025
03/21/2023, 4:26 PMwhat is the timeline for releasing it?