Hi team,
Is there support to create assertions via...
# getting-startedn
nutritious-lighter-88459
07/04/2023, 12:17 PMHi team,
Is there support to create assertions via GraphQL apis?
I see that there is a query
assertion(urn: String!)deleteAssertion(urn: String!)✅ 1
nutritious-lighter-88459
07/05/2023, 11:57 AMAlso, a follow up question is whether we can create Datasets via GraphQL apis ?
nutritious-lighter-88459
07/05/2023, 11:59 AMWe need to evaluate and finalize the APIs that we will use (GraphQL vs OpenAPI) to integrate with datahub. And the evaluation is based on the ability to create and update entities like tags, domains, glossary terms, datasets, data products, ingestion and assertions.
d
delightful-ram-75848
07/06/2023, 6:53 AMHi Ishwar, we don’t support creating dataset via GraphQL at the moment ( cf : https://datahubproject.io/docs/api/tutorials/datasets#create-dataset)
AFAIK, we don’t support creating assertions via graphQL either - @gray-shoe-75895 could you confirm on this?
n
nutritious-lighter-88459
07/06/2023, 12:03 PMHi @delightful-ram-75848, thanks for the response. Any idea when this will be available ?
a
aloof-gpu-11378
07/08/2023, 1:25 AMIt’s possible to create both datasets and assertions using the Python SDK, which uses the Restli APIs under the hood. However, we don’t support those via graphql yet
b
billions-baker-82097
09/22/2023, 6:18 AMHey, just want to follow up on creating assertions and datasets using Python SDK. I was not able to find Python SDK for assertions creation. Can you help me out where to find the code for the same?
a
aloof-gpu-11378
09/22/2023, 11:45 PMWe have a code sample for it here https://github.com/datahub-project/datahub/blob/master/metadata-ingestion/examples/library/data_quality_mcpw_rest.py
👍 1
b
billions-baker-82097
09/24/2023, 6:39 PMThanks! 🙌
billions-baker-82097
09/25/2023, 12:26 PMHi, I also wanted to enable the validation through recipe file for postgres db. Can you point to the example for the same?
Thanks!
a
aloof-gpu-11378
09/26/2023, 1:07 AMWhat exactly are you looking to do?
b
billions-baker-82097
09/28/2023, 5:01 AMYea, so I am inserting metadata through 2 ways postgres db(recipe file) and csv file( Python Sdk), so I want to have Validation tab enabled in both the cases.
a
aloof-gpu-11378
09/28/2023, 5:04 AMYup - if you have data quality stats associated with the dataset urns - either the postgres ones or the ones from your csv file - then the validate tab will be enabled
b
billions-baker-82097
09/28/2023, 6:21 AMCan you give any example of recipe file for postgres to enable the validation tab?
Thanks!
n
nutritious-lighter-88459
09/28/2023, 6:28 AMHi @billions-baker-82097, I believe there is no support currently in Datahub to define data quality checks on a dataset. In our project, we are leveraging Great Expectations to run validations and export the results to Datahub. Which can then be viewed under Validation tab.
👍 1
2 Views