Hi everyone! :blush: I'm new to the community and ...
# community-support
j
Hi everyone! 😊 I'm new to the community and recently started using Elementary for our dbt pipeline. I have a question about optimizing tests. Our process runs every hour on dbt via a Snowpark container, but I'm considering running tests, anomaly detection, and volume alerts just once a day—after midnight—instead of every hour. Previously, I had
dbt build --select elementary
, which ran all the models. After switching to just
dbt build
, I noticed that some Elementary models are no longer running, but the test results are still being imported into the tables. Is there a way to run artifacts every hour while executing anomaly detection less frequently?
n
Hey Jelena, welcome! To check my understanding - you want to run your dbt pipeline every hour without any tests, and test it once every 24 hours. Correct? 🙂
j
Hey Noya, thanks for the reply, yes. I am trying something like this now just want to be sure that the data will be properly picked up for the tests
1. Every hour: dbt build --select tag:bronze --exclude tag:elementary-tests --target prod
2. Once a day: dbt build --select tag:elementary-tests --select elementary --target prod
3. and then edr report --days-back 1 --env prod
n
Got it, thank you! This is my suggestion: 1. Every hour:
dbt build --select tag:bronze --exclude tag:elementary-tests --target prod
. No changes, this will not run Elementary tests but will write run results to Elementary tables. 2. Once a day: •
dbt test --select tag:elementary-tests --target prod
- Will run tests over the existing tables •
edr report --days-back 1 --env prod
Let me know what you think 🙂
j
Aha okay, understood 🙂 Yeah I was not sure about the elementary tables, it makes sense yes. Will try it out and see how it goes for few runs. Thanks a lot Noya!
n
Great! Happy to help.
j
Thanks! I just have one more question for a clarification, please forgive me if it's already answered or if it's an obvious question 😅. When should we use --select elementary? Initially it creates the tables in the elementary schema yes, but when is it needed to be included in runs (upon version upgrades)?
n
This is a very good (and highly confusing) question! Elementary tables are being filled by on-run-end hooks, not by a regular
dbt run
. Because of that, you should use
--select elementary
only on your first Elementary set up, or after a version update in case there's a change in the Elementary table/schema structure.
👍 1
j
Thank you Noya!
n
Hey Jelena, we added some clarification that should help with your question about
--select elementary
🙂
🙌 1