Hi all! I have a question on one of the latest released features, that is the integration of Great Expectations data quality checks in the UI. That feature looks awesome, is there any plan to do the same, but for DBT tests?

@millions-notebook-72121 We plan to add support for dbt test and other frameworks next

We're actually debating whether to use dbt or great expectations as our testing framework. DBT is lighthter weight and we're already up and running with it, GE would require some setup and there is a lot more boilerplate. But, we'd use them in a very similar fashion. It would be a combination of schema tests (check for nulls, types etc) and more business logic tests (for example, check in table ABC there are no records where column X=1 and column Y=2 as these are inconsistent data records). So we'd like to see for each table the percentage of passed tests, and for each column ideally the same and the timeline of failures or pass rate (which is very similar to what was implemented already!)

got it - we will definitely add support for dbt tests. Stay tuned

Sounds super exciting, especially because in the end I think we'll be going for the DBT tests - great expectations is very heavy and does not work very well with Athena (which we're currently using) - also, conditional expectations in GE are a must for us for proper data quality tests, and they are not supported by the SQLAlchemyEngine. I know you have got a million of feature requests and things to prioritise, but would you be able to give an indication of when support for dbt tests will be added? As in, is it something we can expect in the next month, or is it more a 6-month kind of conversation? Related to adding support for dbt tests, I was actually looking at the source code for the Great Expectation actions. Quite a chunky part of that code deals with creating the

assertion_info

to create the

DataHubStdAssertion

. If we use this https://github.com/calogica/dbt-expectations perhaps that part becomes also very simple for DBT as well? We'd still have to write everything else but trying to understand the workings of that code