A complete solution for open data platforms, enterprise data catalogs, data lakes and data management. Open source, mature, fully-featured and production ready.

DataHub

set the channel topic: Channel dedicated to discussing DataHub’s implementation of data contracts, collecting feedback, and setting best practices.

One feedback on Data Contract SLA time specification in the YAML file: Make it a cron entry type, as it might be a repeat schedule and not just a time of the day. If I did not understand something clearly, let me know…

thanks for the suggestion! the example I demo-ed had:
```time: 0700
granularity: DAILY```


I think we will support a few different variants for specifying SLA-s...  with cron being one of the supported syntaxes for sure

set the channel description: Townhall video: <https://youtu.be/KHNPjSbbZR8?t=2752>

Hi!

I recently found this PR <https://github.com/datahub-project/datahub/pull/7668> that proposes Schema as top level entity.
IMO it may overlap with Data Contracts, which somehow also makes schema first-class citizen while including other stuff, such as SLAs. However, this other stuff (_eg_ SLAs) is directly related to a Dataset. So it constraints a 1:1 relationship between Data Contract and Dataset (that was my impression from the townhall video, please correct me if I'm wrong).

In my organization we have plenty of JSON schemas (tracking tagging plans) and AVRO schemas (kafka events) which may be shared by many datasets. That's we were considering having _Schema_ as top level entity (that's why I found above PR :sweat_smile:). So data catalogue also becomes schema catalogue, for discovery and governance ofc.

What's your view on this?

<@U027ZS25RFS> Do those schemas tied up to dataset somehow? In my past job i find it very helpful to have datahub integrated to your build pipeline so you can act i.e. notify consumers when the schema/contract of a dataset changes.

Yes, but one schema may be used by many datasets.
That's why we are trying to promote Schema as a top-level entity.

Hey <@U027ZS25RFS> schema as top level entity makes complete sense and is not in conflict with contract including a schema 

Specifically the schema specification in a contract can represent just the enforceable piece of the contract (Eg schema type and field name etc). 

Makes sense, yes. Then, is there any plan to retake the work in the PR?

Absolutely! Waiting for schema registry removal to be complete, before taking this on.

I am doing a POC on DataContract but found the DataHub documentation is not yet updated for DataContract command (in CLI) or for APIs (OpenAPI). When can we expect these and in which version?

Hey <@U05A8LBFNP4>: This will be released in the open source code base by late June (latest by early July)

Hi <@UV0M2EB8Q>, we wanted to have a look on the data contract but also couldn't find it in the doc yet. when is the release for the open source version planned?

Hey  <@U0618RW42Q1>  , we don't have a good material on data contract on our doc-site yet, but this blog post will help you to understand the data contract on our oss / sass datahub. :)

Thanks for the answer <@U04QRNY4ZHA>... Did you want to post the link to the blog?

<@U0618RW42Q1>: here's the link: <https://www.acryldata.io/blog/data-contracts-in-datahub-combining-verifiability-with-holistic-data-management>

Hi <@UV0M2EB8Q>, nice post and approach to data contracts!
I'm just wondering about 'Non-Verifiable Metadata'. Is all metadata not verifiable in some way, e.g. a data product should always have a data owner (present / not present). Same for documentation. DP should have at least 3 tags (for example). And a glossary term should be present in a defined Business Glossary? Interested in the community's thoughts.

hi, i’m especially interested in data contracts! would be interested to hear more about their lifecycle and how we are supposed to be integrating with this feature

We’re actively developing our thinking around this right now, but here’s some thoughts
• Ideally data contracts are defined in git source control (using the yaml spec demo’d at the May town hall) or in datahub
• The process of writing contracts is a collaborative process between producers and consumers - ideally that collaboration happens on github PRs or in datahub too, so that the context can be preserved
• Once a data contract is “committed”, DataHub should be responsible for keeping the monitoring systems (e.g. assertion evaluators like dbt / great expectations) updated and collecting results

We’re building out CLI tools for (1) keeping the yaml and datahub in sync (demoed in May) and (2) keeping assertion evaluator systems in sync (demoed today). Together, these tools should help you manage the full data contract lifecycle

<@U01GZEETMEZ> HI
Can we view it on the release or demo stand?

Don’t think we have any public demos for this just yet

also, was wondering how data contracts could be added on to the lineage to visually show where data products responsibilities start and finish. at the moment we got column level and table level lineage which can be overwhelming. Believe having some sort of data product lineage combined with data contracts would clearly show the whole’s company’s data landscape at a glance

Hi all. When did you plan to open source release with data contracts?

has renamed the channel from "talk-data-contracts" to "advice-data-contracts"

Hi All! Can any of you legends advise when data contracts and the integration to raise DBT PRs via data contracts will be available in the open source version?

<@U01GCJKA8P9>  might be able to speak to this!

I can't made a hard promise here, but we're targeting the full data contracts UI support in open-source by mid Nov, and the dbt contracts integration will be made open-source in the weeks following that

In the interim, I'd love to better understand what you're thinking about using data contracts for, and how datahub and dbt fit into that story

Hi <@U01GZEETMEZ> - any update on the timeline above?
I am looking forward to see data contracts UI support in open-source.