https://datahubproject.io logo
Join Slack
Powered by
Hi folks has anyone seen issues with postgresql in...
# ingestion
c
Hi folks has anyone seen issues with postgresql ingestion? I am getting this error when i enable profiling
Copy code
215      if not sqlalchemy:
--> 216          raise DatasourceInitializationError(
    217              name, "ModuleNotFoundError: No module named 'sqlalchemy'"
    ..................................................
     self = <great_expectations.datasource.sqlalchemy_datasource.SqlAlchemyDatasource object at 0x13624d2b0>
     name = 'my_sqlalchemy_datasource-bfb77e2d-9b3d-4e9b-a714-4675fa7c6f08'
     data_context = None
     data_asset_type = {'module_name': 'great_expectations.dataset',
                        'class_name': 'SqlAlchemyDataset'}
     credentials = {'url': <postgresql+psycopg2://username>:***@localhost:5432/databasename}
     batch_kwargs_generators = None
     kwargs = {'engine': <sqlalchemy.engine.base.Connection object at 0x13620c400>}
     sqlalchemy = None
     DatasourceInitializationError = <class 'great_expectations.exceptions.exceptions.DatasourceInitializationError'>
    ..................................................

---- (full traceback above) ----
File "/Users/ankur.chauhan/.pyenv/versions/3.9.5/lib/python3.9/site-packages/great_expectations/data_context/data_context.py", line 1869, in _instantiate_datasource_from_config
    ] = self._build_datasource_from_config(name=name, config=config)
File "/Users/ankur.chauhan/.pyenv/versions/3.9.5/lib/python3.9/site-packages/great_expectations/data_context/data_context.py", line 1935, in _build_datasource_from_config
    datasource = instantiate_class_from_config(
File "/Users/ankur.chauhan/.pyenv/versions/3.9.5/lib/python3.9/site-packages/great_expectations/data_context/util.py", line 121, in instantiate_class_from_config
    class_instance = class_(**config_with_defaults)
File "/Users/ankur.chauhan/.pyenv/versions/3.9.5/lib/python3.9/site-packages/datahub/ingestion/source/ge_data_profiler.py", line 64, in sqlalchemy_datasource_init
    underlying_datasource_init(self, *args, **kwargs, engine=conn)
File "/Users/ankur.chauhan/.pyenv/versions/3.9.5/lib/python3.9/site-packages/great_expectations/datasource/sqlalchemy_datasource.py", line 216, in __init__
    raise DatasourceInitializationError(

DatasourceInitializationError: Cannot initialize datasource my_sqlalchemy_datasource-bfb77e2d-9b3d-4e9b-a714-4675fa7c6f08, error: ModuleNotFoundError: No module named 'sqlalchemy'
i
Hello Ankur, It looks like a dependency of the ingestion framework was not installed before running the ingestion.
How are you running this ingestion? Docker? Python cli?
c
i am using python cli
datahub ingest -c <….>
i
Did you install the CLI via PyPI?
Can you check if you have SQLAlchemy installed?
Copy code
$ pip list | grep SQLAlchemy
Should return the version of the lib if it's installed.
c
Copy code
$ pip list | grep SQLAlchemy
Flask-SQLAlchemy                  2.5.1
SQLAlchemy                        1.3.24
i installed the dependencies via pip
i
Before running the ingestion?
Since you're running the postgres ingestion, please make sure to check the associated setup: https://datahubproject.io/docs/metadata-ingestion/source_docs/postgres
run: 
pip install 'acryl-datahub[postgres]'
c
Tried again --- no dice. same error.
g
@calm-airplane-47634 I believe this is an issue with one of our dependencies, great expectations. See https://github.com/linkedin/datahub/pull/3634 for details - if you run 
pip install --upgrade 'acryl-datahub[postgres]' great-expectations
it should do the trick
👀 1
c
That’s it! thankyou
2 Views
Open in Slack
PreviousNext
Powered by