A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

Hi we're having troubles with Postgres ingests, they fail with error `"ImportError: cannot import name 'Distribution' from 'pkg_resources' (unknown location)\n",` . Any idea why that might be happening?
They used to ingest successfully before, and the failes started about two weeks ago or so

Hi <@U03LYB2ESJ0>, what version are you on, and have you recently upgraded?

now we're on 0.9.6, but the errors started on 0.9.5. on which we upgraded in december.

<@U042KCFV9GX> ok from what I got from googling (git <https://github.com/googleapis/python-api-core/issues/27|link>), looks like pkg_resources is known to cause such issues and is 'semi-deprecated and are mostly unsupported'. Is there a reason why Datahub is using it and is there a way to work around using it in ingests?

<@U03LYB2ESJ0>, it’s not us, but one of our dependencies- could you post some of the logs from the failed ingestion runs?

exec-urn_li_dataHubExecutionRequest_5c1e0a17-b9ed-40d3-bbef-b7164c392d29.log

Ok thanks! CC: <@U01GZEETMEZ> seems like a GE dependency issue

Looks like this line <https://github.com/great-expectations/great_expectations/blob/develop/great_expectations/util.py#L55> has been in GE for a while. My guess is this is somehow using a new/old of setuptools, but unfortunately I don’t think there’s a ton that we can do

<@U01GZEETMEZ> but now it intervenes all our postgres ingests with profiling - they all fail with this error (but ingest succussfully if I disable profiling). Profiling is a part of basic Datahub functionality in my understanding, is there really nothing that can be done?

I just tested postgres ingestion with profiling enabled and it seems to have worked ok on my end. Are you on the latest datahub version, and do you have any “cli version” set in the advanced part of your config?

<@U01GZEETMEZ> hmmm, that's strange. We're on 0.9.6 and we don't have and cli version set in advanced configs. What else can I check? Maybe we should specify a cli version?

<@U01GZEETMEZ> I set the 0.9.6 CLI version in the advanced part and it worked, thank you! I the only question that remains now is why Datahub doesn't automatically use CLI version in accord with the release version, but uses an old one, which leads to errors:thinking_face:

That is very strange - we should be keeping them aligned by default so the misalignment is probably a bug - I’ll look into it a bit further