Export dataset validation results to BI dashboard (e.g. PowerBI) We were asked by our client whether it'd be possible to create a dashboard with DQ checks history in a BI tool. Just wondering if anyone had similar task before and what would be the simplest way to accomplish this. API requests directly from PowerBI using GraphQL API? Or maybe sync some underlying RDS structures (we're on AWS) to Snowflake and then create a dashboard on top of it? The sync does not have to be real-time, even once a day is sufficient. Thanks in advance for the tips!

hey there! just want to make sure I have the ask correct - so you have some upstream datasets of a BI dashboard and you want to get those assertions from those datasets onto the dashboard in PowerBI? or you want to create a dashboard of different dataset assertions?

The case is like this: • I have a pipeline with dbt tests consisting of assertions that are run along with the model in order to make DQ checks • These test results are then synced to DataHub (source: Snowflake) and visible in

Validation

tab of a tested dataset • As a business requirement I need to embed tests for some of the datasets in a PowerBI dashboard (imagine it as a one-pager that tells what's the condition of the data for business stakeholders) • For this reason I need to get access to where DataHub stores these tests - either via API directly or by syncing these tables to Snowflake before visualising them in PowerBI

okay gotcha! that makes total sense, and thanks for explaining further. I'm not entirely sure how it would work/look if you were to sync to Snowflake before visualizing them, but you can certainly pull these assertions for the datasets you care about via the GraphQL API! What you'll need is the urn of the dataset you care about, and you can issue a query like this:

query getDatasetAssertions($urn: String!) {
    dataset(urn: $urn) {
        urn
        assertions(start: 0, count: 100) {
            start
            count
            total
            assertions {
                runEvents(status: COMPLETE, limit: 1) {
                    total
                    failed
                    succeeded
                    runEvents {
                        status
                    }
                }
            }
        }
    }
}

or you can issue a similar one for a list of datasets if you care about more than one:

query getEntities($urns: [String!]!) {
    entities(urns: $urns) {
        urn
        type
        ... on Dataset {
            assertions(start: 0, count: 100) {
                start
                count
                total
                assertions {
                    runEvents(status: COMPLETE, limit: 1) {
                        total
                        failed
                        succeeded
                        runEvents {
                            status
                        }
                    }
                }
            }
        }
    }
}

Thanks for the tips! Initially I thought that I'd be skipping the API in order to allow business users easily define what they want to retrieve from metadata (hence this idea with querying RDS/MySQL directly). But now I realized that it might not be safe as the structures in the db might change while the API should be more stable... There are still some unknowns for me in terms of how the PowerBI dashboard would be easy to configure - i.e. if there is a chance to parametrize something without touching the underlying code, but I think I'll give it a go. Thanks once again!

yeah that all makes sense to me! I think giving the API a go and seeing how that all works out would be a good next step. please let us know how it's going as you continue on your project!