Hello, I just saw in our Cloud build logs that in ...
# community-support
c
Hello, I just saw in our Cloud build logs that in our daily load a elementary model dbt_models takes up huge amount of time. It merges 0 rows but take 3350 seconds ( 55minutes) which is beyond acceptable. What is going on here?
✅ 1
Same thing with running edr monitor for alerts, this is in our dev environment that holds 86 resources (models/tests) which doesn't contain almost any data:
Anyone?
@Ofek Weiss @Elon Gliksberg do you know?
e
Hey there @Christopher Ekfeldt! What version of Elementary and Dbt are you running? If you could, add all that info to a Github issue so we can track and make sure it's dealt with.
👍 1
c
We are using elementary 0.13 with dbt 1.7.0
I will create a Github Issue.
🐕 1
@Erik Zaadi what is dbt_columns used for ? Could we disable it using: disable_dbt_columns_autoupload: true ? And how will that affect us.
e
We gather that for our artifacts, but it should not affect your reports. We'll still try to fix the speed of that as fast as possible, however the flag is there to overcome your current problem
👍 1
@Christopher Ekfeldt let me know if it helped with the run time
c
It doesn't work, This is our current config:
elementary:
+dataset: "elementary"
+enabled: "{{ not target.name.endswith('_ci') }}"
disable_run_results: "{{ target.name != 'prd' }}"
disable_tests_results: "{{ target.name != 'prd' }}"
disable_dbt_columns_autoupload: true
disable_dbt_artifacts_autoupload: true
disable_dbt_invocation_autoupload: "{{ target.name != 'prd' }}"
collect_metrics: "{{ target.name == 'prd' }}"
In our top 15 most heavy models , elementary is 3 of them. dbt_columns takes ~44 minutes in our daily run. dbt_tests take ~18 minutes in our daily run. dbt_models take ~12 minutes in our daily run. What is going on, are we doing something incorrectly?
So dbt_columns still runs for 44 minutes with "disable_dbt_columns_autoupload: true" being set.
o
are all the variables set under
vars
(from what you send it does not seem like the case), a screenshot of your
dbt_project.yml
file would be helpful also, can you share your
dbt.log
file?
c
Yes, they are set under the vars. Our dbt_project.yml is huge, so I just copied the necessary configs. 🙂 Don't think I can share our dbt.logs , not here at least.
o
you can do so privately if you prefer
✅ 1
j
Did you manage to find a resolution to this as we are having the same problem 😭