<@U02QGCRKQ22> Hi!
Seems like lots of people here...
# integrate-iceberg-datahubf
full-alligator-99452
12/21/2023, 12:30 PM@modern-monitor-81461 Hi!
Seems like lots of people here are facing the same issue with integration between DataHub and Apache Iceberg using SQL or Hive meta catalog. As I understood, in the end of summer pyiceberg library was updated to 0.4.0 for DataHub and, I guess that connection should be possible at least with Hive Met Catalog.
Would you be able please to let us know if you have any suggestions how we can resolve that error:
“Apache Hive support are not installed”?
I was able to “pip install pyicberg[hive]==0.4.0” on server itself and into docker container “DataHub-actions” if it makes sense, but error remains the same.
full-alligator-99452
12/21/2023, 5:37 PMJust FYI, if someone will be searching for the same question. I just added thrift library to specific destination in datahub-actions docker container and it worked.
pip install thrift --target=/tmp/datahub/ingest/venv-iceberg-f3c1b67e57e34548/lib/python3.10/site-packages
v
victorious-car-1170
12/28/2023, 11:03 AMHi @full-alligator-99452 thanks for suggestion. Have you able to ingest data of iceberg using Hive catalog.
victorious-car-1170
12/28/2023, 11:04 AMI was also trying for same, any extra options you tried to success
victorious-car-1170
12/28/2023, 11:25 AMEven for me @lively-appointment-50242 @full-alligator-99452 I have used hive catalog, somehow ingestion showing Failed. But the metadata is populating in datahub. Very Strange. In Ingestion logs it says below error
Copy code
'failures': {'general': ['Failed to create workunit: Property table_type missing, could not determine type: sf1000delta.store_sales', 'victorious-car-1170
12/28/2023, 11:25 AMThanks to both of you for insight on solution @lively-appointment-50242 @full-alligator-99452 👍
👍 1
f
full-alligator-99452
01/08/2024, 11:39 AM@victorious-car-1170 Hi!
Was on a short vacation :)
Yes, I was able to ingest metadata from iceberg with hive meta catalog. Even in ingestion tasks have status “success”.
As mentioned, I’ve just added that “thrift” library in specific folder inside “datahub-actions” docker container. Haven’t done anything else special.
👍 1
v
victorious-car-1170
01/08/2024, 12:05 PMYeah, did the same thing, but somehow my ingestion status is failed, but metadeta injestion happening. thanks
victorious-car-1170
01/08/2024, 12:05 PMDid you tried starburst integration with datahub any time