Hi Team, I would like to reframe the same question in my previous message if it was not clear enough.
So, we are trying to integrate Datahub with Great Expectations and in this process, we have set up the expectations suite, stored the validation results, added the checkpoint to the snowflake config context, and ran the same to see if the results were appearing in the validation tab of datahub for that particular table.
Initially, we have done this on the datahub version which is running on Azure Kubernetes services but we had a doubt if this would succeed as this is a cloud environment. So, we started our work on a datahub hosted on a local server, but the ingestion is not successful either way. As I mentioned earlier below things are in place to get this job done, please let us know if we are something
1. Context has been created with Snowflake connection details for the table to validate.
Expectation Suite is created, Rules are added to Suite and saved the same.
2. Checkpoint is also added to the context
batch_request pointing to the Snowflake Table to be Validated and listing actions information with the "datahub.integrations.great_expectations.action" class and
Datahub Local Host connection.
3. Finally, validator and run_checkpoint are called on top of the context object to complete the work.
Having done all these, we weren't able to find the expected result in Datahub. We are running this through Python and it is returning "exit Code 0"
Kindly let us know if anyone has an idea of where it was going wrong. I have mentioned a few code lines below in the sequence we have executed. In depth, if anything is required please let us know, and I’ll share in detail.
Code:
Creating Context :
data_context_config = DataContextConfig(
datasources={
"sqlalchemy_datasource": DatasourceConfig(
class_name="Datasource",
execution_engine={
"class_name": "SqlAlchemyExecutionEngine",
"connection_string": f<snowflake connection>
},
data_connectors={
"runtime_data_connector": {
"class_name": "RuntimeDataConnector",
"batch_identifiers": [
"customerid_key_0",
"customerid_key_1"],
}
}
)
},
expectations_store_name="expectations_store",
validations_store_name="validations_store",
evaluation_parameter_store_name="evaluation_parameter_store",
checkpoint_store_name="checkpoint_store",
store_backend_defaults=InMemoryStoreBackendDefaults(),
)
context = gx.get_context(project_config=data_context_config)
context.create_expectation_suite(expectation_suite_name=<suite name>)
suite.add_expectation_configurations(rules)
context.save_expectation_suite(expectation_suite=suite, expectation_suite_name=group_name)
context.add_checkpoint
context.get_validator(batch_request=, expectation_suite=suite)
context.run_checkpoint