Hello. We are planning to evaluate datahub entities (mainly datasets) against company compliance rules which could be reported back to the dataset owners. When I think about it that seems very similar to the ingestion pipeline but reversed such that the source would be datahub, the transformers would be the compliance rules and the output a db, console, api etc. Has anyone else done something like this already? Would it make sense to make a feature request?

Interesting! What kind of rules are you thinking about?

Could be anything but basic examples: • Missing basic documentation • Missing owner • Missing schema

Ok, makes sense. We were planning to do some of that already at ingestion stage (basically forcing the existence of the metadata in the source as well), due to lack of the feature you're describing.

Yea, that makes sense as an alternative.

@wonderful-quill-11255 Yep there’s a couple ways I can imagine doing this: (1) fetching data via the graphql interfaces (or possibly from GMS) and running checks (2) fetching data directly from datahub’s sql store and running scripts on that (3) subscribing to the MAE kafka topic and running validations as the metadata changes. On (2) - this is actually what our demo is doing right now! We have a backup of the core mysql table in S3, which is processed by an airflow pipeline and then fed into superset to produce charts like “dataset documentation coverage by platform” etc

Yup, but then I need logic to only use the latest version of each aspect for example.

Yup, but then I need logic to only use the latest version of each aspect for example.

This specific one is just adding a WHERE version=0 clause, but using the sql tables might be cumbersome due to the json nesting I’d personally recommend using the graphql interfaces that the frontend provides, since it can be strongly typed and the codegen tooling around graphql is quite good, but ofc it depends on the exact use cases

Ok. Any thoughts on whether a library or framework for this use-case is worthy of a feature request?

Yep I definitely think so. We should probably start with a set of scripts/recipes about how to validate against common rules (e.g. missing owners), and then build up to a proper framework

@wonderful-quill-11255 Being able to take action based on Metadata changes will definitely be a big part of our H2 roadmap. There are two aspects to your requirement: 1) How to run the compliance validation checks when entities get modified (e.g: new dataset added, schema changes. new PII tag added on a column etc.) 2) Running these checks in batch

Sounds good. I'll PM you.