Hi all! Our team is implementing Elementary on our...
# generalm
Mariah Rogers
04/19/2022, 10:53 PMHi all! Our team is implementing Elementary on our dbt project. I was wondering if there was a way to enable the elementary configurations and default tests on entire schemas / folders of models rather than having to copy the same configurations into every individual model
ymlo
Or Avidov
04/20/2022, 8:38 AMHi @Mariah Rogers! 👋 currently the monitors are defined as tests and unfortunately dbt doesn't support at the moment a way to define tests globally on entire schemas or folders. Another issue with defining monitors globally is that it's really recommended from performance perspective to define a timestamp_column that is used for both freshness calculations and limiting the amount of data that is being scanned.
Or Avidov
04/20/2022, 8:47 AMA possible workaround that was suggested in the dbt's community was to use YAML anchors to possibly reduce the configuration work -
Or Avidov
04/20/2022, 8:48 AMThat being said, we are actively researching creative ways to better support this use case and would love to explore them with you 🙂. A followup question we had about it - do you have a generic name for your timestamp columns across different tables / models?
j
Jake Beresford
04/20/2022, 1:20 PM@Mariah Rogers This was the original thread in dbt slack https://getdbt.slack.com/archives/C2JRRQDTL/p1649955100718789
Jake Beresford
04/20/2022, 1:22 PM@Or Avidov Mariah is a colleague of mine 🙂 - the vast majority of our timestamps are
updated_atJake Beresford
04/20/2022, 1:29 PMThinking along the same lines, what if elementary tested all models by default with the ability to disable or configure a subset of monitors as configuration? Switching the config to opt-out instead of opt-in would reduce initial setup time-cost, though it might be a little unconventional
o
Or Avidov
04/20/2022, 2:14 PMYeah it makes sense. We have an idea on how this could be done using a new type of test that will run on multiple tables instead of one table / column. We could create a global model in our package and you can create its yml file and define these new tests on this model. The main downside would be that when there is an issue with one of the tables / columns this global test will fail and you will only be able to understand which table / column has an issue in the slack alerts / alerts table that is being saved in the DWH
🤔 1
Or Avidov
04/20/2022, 2:25 PMWe can mitigate this downside with logs by printing to the screen while running the tests which tables / columns are problematic
33 Views