Hi all, I am looking at upgrading the Datahub Iceberg source to the new Iceberg Python SDK (so we can get rid of https://github.com/acryldata/py-iceberg). I have just added support for Azure Datalake in the new pyiceberg, so migrating to this new package brings us on par with what we have right now. But it sets the Iceberg source in position to fully support Iceberg and all the catalogs once those are added to pyiceberg (currently being worked on by the Iceberg team). I have refactored the Datahub Iceberg ingest source code, but I'm running into package version issues related to pydantic:

ERROR: Cannot install acryl-datahub[dev]==0.0.0.dev0 and pyiceberg==0.2.0 because these package versions have conflicting dependencies.

The conflict is caused by:
    acryl-datahub[dev] 0.0.0.dev0 depends on pydantic>=1.5.1
    acryl-datahub[dev] 0.0.0.dev0 depends on pydantic<1.10 and >=1.9.0; extra == "dev"
    acryl-datahub[dev] 0.0.0.dev0 depends on pydantic>=1.5.1; extra == "dev"
    pyiceberg 0.2.0 depends on pydantic==1.10.2

pyiceberg requires pydantic 1.10.2, but DataHub seems to have a type issue with 1.10+ according to this comment. What is this about? Is it something we can fix? @gray-shoe-75895