Hello Team Has anybody here worked on Datahub's in...
# ingestion
m
Hello Team Has anybody here worked on Datahub's integration with great-expectations : i.e pushing the validation results of great expectations to datahub for a CSV /Parquet file. I could only accomplish this for SQL alike data sources (i.e BigQuery)
m
@hundreds-photographer-13496 might have answers for you here
h
Hey @mysterious-pager-59554 my answer is in fact exactly same as the answer by maggie - https://datahubspace.slack.com/archives/C02FD9PLCA0/p1658949313526859?thread_ts=1658931374.854149&cid=C02FD9PLCA0 If you would like to add this feature yourself, I would be very glad to help. We can discuss in #contribute channel for any help required. I have some questions for you, to understand the scope of your usecase. 1. Which execution engine / datasource are you using for validating csv/parquet? 2. Where are csv/ parquet files that are being validated stored (local file system, s3 , etc)? 3. Have you already ingested csv/parquet files as datasets in DataHub ?