I have seen similar posts, but I am trying to understand why

on-run-end

hook for elementary takes so long. For reference, running a set of models with specific tags (128 models out of 370, the rest are views) takes 2 and a half minutes, while running the hook for the same set takes 7 minutes. I checked the log, I don't see any operation exceeding 4 seconds, I also tried to disable the artifacts autoupload, but no improvements. I am not sure if the number of tests is the issue (6723), but also I am testing only on

dbt run

commands, so it should not touch the test tables in the end of the run right? This is how the variables are configured

disable_dbt_artifacts_autoupload: "{{ target.role != 'PROD_RW_ROLE' }}"
  disable_run_results: "{{ target.role != 'PROD_RW_ROLE' }}"
  disable_tests_results: "{{ target.role != 'PROD_RW_ROLE' }}"
  disable_dbt_invocation_autoupload: "{{ target.role != 'PROD_RW_ROLE' }}"

so it runs only when this role is used. We are on Snowflake

I find it takes awhile on BigQuery as well.

I investigated a bit more, and the culprit seems to be the tests (of course)

2 of 11 OK created sql incremental model ELEMENTARY.dbt_invocations ......[32mSUCCESS in 5.40s]
5 of 11 OK created sql incremental model ELEMENTARY.dbt_run_results ......[32mSUCCESS in 6.17s]
3 of 11 OK created sql incremental model ELEMENTARY.dbt_metrics ..........[32mSUCCESS in 6.59s]
7 of 11 OK created sql incremental model ELEMENTARY.dbt_snapshots ........[32mSUCCESS in 8.41s]
6 of 11 OK created sql incremental model ELEMENTARY.dbt_seeds ............[32mSUCCESS in 9.26s]
1 of 11 OK created sql incremental model ELEMENTARY.dbt_exposures ........[32mSUCCESS in 10.17s]
8 of 11 OK created sql incremental model ELEMENTARY.dbt_sources ..........[32mSUCCESS in 20.74s]
4 of 11 OK created sql incremental model ELEMENTARY.dbt_models ...........[32mSUCCESS in 31.87s]
10 of 11 OK created sql view model ELEMENTARY.dbt_columns ................[32mSUCCESS in 26.16s]
9 of 11 OK created sql incremental model ELEMENTARY.dbt_tests ............[32mSUCCESS in 370.78s]
11 of 11 OK created sql view model ELEMENTARY.dbt_artifacts_hashes .......[32mSUCCESS in 0.52s]

but this still does not explain why it takes so long even if I don't run any tests

@Igor Dal Bo do you set

disable_dbt_artifacts_autoupload

to

false

? Once you've done that, when are you running

dbt run --select elementary.edr.dbt_artifacts

? Before you build your elementary report and alerts, or is it less frequent than that?

so this is what I have under

vars

in the

dbt_project.yml

# artifacts are fully disabled by default
  disable_dbt_artifacts_autoupload: true

and we only upload the artifacts when we deploy (I thought of any possible case, and for us this is the only situation where artifacts can change), in the script that deploys we run

# update artifacts manually
echo "ok - uploading artifacts"
dbt run --select elementary.edr.dbt_artifacts

after

dbt run dbt test dbt snapshot dbt seed

(basically it is the last step of deployment)