Hi Everyone, Trying to use Great expectation for Data Validation. The checkpoint runs, but the Validations are not getting displayed in Datahub. Anything that I need to take care of expect the Datahub Action Added this is in checkpoint configuration

- name: datahub_action
  action:
    module_name: datahub.integrations.great_expectations.action
    class_name: DataHubValidationAction
    server_url: <https://hostname>

Getting message when checkpoint runs

great_expectations checkpoint run postgres_checkpoint
Using v3 (Batch Request) API
Calculating Metrics: 0it [00:00, ?it/s]
WARNING: Enable parse_table_names_from_sql in DatahubValidationAction config              to try to parse the tables being asserted from SQL query
Validation succeeded!

Suite Name                  Status   Expectations met
- public.tablename.suite      ✔ Passed  0 of 0 (100 %)

cc: @green-football-43791

Hello @handsome-football-66174, from logs, it seems that you are using sql query for forming batch. Reporting validations on tables from sql query is not enabled by default, hence the warning message. Can you set

parse_table_names_from_sql

config to

True

and retry ?

- name: datahub_action
  action:
    module_name: datahub.integrations.great_expectations.action
    class_name: DataHubValidationAction
    server_url: <https://hostname>
    parse_table_names_from_sql: True

Also if query is formed using more than one tables, the validation results are currently not reported to datahub.

@hundreds-photographer-13496 - I enabled it and executed. But the validation results are not visible ( currently pulling only from one table )

Can you enable INFO level logs and confirm if you see any log mentioning "Metadata not sent to datahub. No datasets found." ?

@hundreds-photographer-13496 - I dont see the logs similar to what you have described. Also how do I enable INFO level logs in great_expectations

yes, DataHubValidationAction supports only the datasources using SqlalchemyExecutionEngine as of now , also mentioned in doc https://datahubproject.io/docs/metadata-ingestion/integration_docs/great-expectations/#capabilities Please show your interest by upvoting this datahub feature request if you would like DatahubValidationAction to work with pandas/spark execution engine (which is typically used to connect with s3/glue). https://feature-requests.datahubproject.io/b/feedback/p/great-expectations-support-different-execution-engines

@hundreds-photographer-13496 - For Tables present in Glue Catalog ( and have been ingested in Datahub with Glue as source ) , can we validate the data using Athena ? Or do we need to ingest using Athena as source and then validating using great expectations

If data is validated for Athena datasource in great expectations, that would ingest assertions against Athena dataset in datahub as opposed to glue/S3 dataset. So I believe you would need to ingest Athena as source to be able to see validations. Let me know if that answers your question.

@hundreds-photographer-13496 - Getting the following error when trying to validate data in Athena using GE ( expectations etc are executing , but is unable to emit to datahub) -

ERROR: ('Unable to emit metadata to DataHub GMS', {'message': "HTTPSConnectionPool(host='hostname', port=443): Max retries exceeded with url: /aspects?action=ingestProposal (Caused by SSLError(SSLEOFError(8, 'EOF occurred in violation of protocol (_ssl.c:1125)')))"})

Seems to be an SSL error - not directly related to datahub. I don't have much familiarity with it to help you out. Have you tried any troubleshooting guide online ?

@hundreds-photographer-13496 - When I try to ingest tables from Athena , I am getting this warning , pipeline completes , but no dataset created in UI

AIRFLOW_CTX_DAG_RUN_ID=manual__2022-05-13T14:25:40.405498+00:00
[2022-05-13 14:25:44,446] {logging_mixin.py:104} INFO -  Pipeline config is {'sink': {'type': 'datahub-rest', 'config': {'server': 'https://<gms>'}}, 'source': {'type': 'athena', 'config': {'env': 'PROD', 'aws_region': 'us-east-1', 'work_group': 'primary', 's3_staging_dir': '<s3 location>', 'database': 'covid_19', 'table_pattern': {'allow': ['tablename*'], 'ignoreCase': 'false'}}}, 'transformers': [{'type': 'simple_add_dataset_ownership', 'config': {'owner_urns': ['urn:li:corpuser:<user>']}}, {'type': 'set_dataset_browse_path', 'config': {'path_templates': ['/PLATFORM/DATASET_PARTS']}}]}
[2022-05-13 14:25:45,612] {logging_mixin.py:104} WARNING - /home/ec2-user/.venvs/airflow/lib64/python3.7/site-packages/snowflake/connector/options.py:97 UserWarning: You have an incompatible version of 'pyarrow' installed (8.0.0), please install a version that adheres to: 'pyarrow<5.1.0,>=5.0.0; extra == "pandas"'
[2022-05-13 14:27:36,340] {_lineage_core.py:295} INFO - DataHub lineage backend - emitting metadata:

@hundreds-photographer-13496 We are having the same problem Every think is working but the datahub did not activate the validation .... and the logs of gms are the same as what you have mentioned

@plain-napkin-77279 - Unfortunately havent resolved it yet.

Actual problem! Please support the feature request