Looking for best practice tips on setting up tests...
# community-supportd
Damian Eralio
10/09/2023, 3:00 PMLooking for best practice tips on setting up tests / alerts to strike a balance between thorough testing and avoiding alert fatigue.
Our setup is source -> staging -> intermediate (optional for joins) -> marts
Is it recommended to set up table level tests on sources rather than staging models? both?
a
Alex
10/09/2023, 3:59 PMThat is a very good question Damian, and there is no 1 fits all solution.
I have these 2 principles when deciding where to implement tests:
• If a data quality issue can cause a significant business impact, the right people should be alerted.
• If nobody acts when an error alert occurs, the alert is useless.
Alex
10/09/2023, 4:03 PMSo I may have tests in all layers.
I have a project where the staging layer is a 1:1 model (materialised as view) with the sources. The staging models remove deleted rows (marked as deleted by the EL tool), rename and cast columns. (No joins).
So I put the source tests in the staging layer as I am not using the default dbt source freshness (I use elementary freshness instead).
Alex
10/09/2023, 4:06 PMYou should test joins during development as it is easy to generate duplicates because of a wrong join for example.
But in production you may decide not to test them again if the tables are too big and the queries too expensive.
In the mentioned project, the mart layer had several joins, the dataset was small and it was a fintech company. So we left in production some key tests that should never ever fail.