Hi everyone, the on-run-end hook is significantly slow in our use case with dbt-bigquery. Is there any ways to speed up it? It takes more than 20 minutes to insert results to the datasets for Elementary on BigQuery. The longest process of the on-run-end hook took 75 minutes in our case. • dbt-core: 1.7.3 • dbt-biquery: 1.7.6 • elementary dbt Package: 0.13.2

03:19:52  Running 1 on-run-end hook
03:43:48  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
03:43:48  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]

Following. Same issue here...

Following, i dont even finish the post hook, it just failed. and all the inserts are null..

I don’t exactly know the configurations. Can we take advantage of them? Apart from that, other properties looks available to tune Elementary. https://github.com/elementary-data/dbt-data-reliability/blob/master/macros/edr/system/system_utils/get_config_var.sql#L22-L66

# dbt_project.yml
vars:
  disable_run_results: "{{ target.name not in ['prod','analytics'] }}"
  disable_tests_results: "{{ target.name != 'prod' }}"
  disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"
  disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"

I’m trying the properties, as macros related to them took a long time in our case. However, that would be good, if

disable_dbt_artifacts_autoupload

doesn’t affect to use the community edition of Elementary, not Elementary Cloud.

upload_artifacts_method: replace
disable_dbt_columns_autoupload: true

hi @yu-iskw @Noam Moskowitz @Adam, can you share your

dbt.log

of these long runs?

I wanted to share some similar sluggish behavior: We’re on

dbt-bigquery==1.7.4

and we recently upgraded our package from

elementary-data/elementary==0.11.2

to

elementary-data/elementary==0.13.2

. On

0.11.2

, the post-run hook took less than a second:

00:01:44
00:01:44  Running 1 on-run-start hook
00:01:44 1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
00:01:44 1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
00:01:44

On

0.13.2

, the post-run hook takes nearly a minute (note the leftmost timestamps, rather than the

[OK in 0.00s]

message:

00:34:59
00:34:59  Running 1 on-run-end hook
00:35:52 1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
00:35:52 1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
00:35:52

Looking at our BigQuery audit logs, it appears that the Elementary

on-run-end

hook is inserting into the following tables in our Elementary dataset (I’ve included the job duration in parentheses): •

data_monitoring_metrics

(1 sec) •

dbt_artifacts_hashes

(1 sec) • 2 instances of a test that runs

SELECT metadata_hash from dbt_tests WHERE 1=0

🤷 (1 sec) • 2 instances of inserts into

dbt_tests__tmp_*

(1 sec) •

dbt_tests

(5 sec) •

dbt_run_results

(1 sec) •

elementary_test_results

(1 sec) •

dbt_invocations

(1 sec) So, 12 seconds of BigQuery job time in total.

Thanks @Brian Waligorski That’s interesting, to understand if we’re having similar use cases here can you all also give us your warehouse type (bigQuery, snowflake, etc) and what versions you’re on? (package, cli, dbt) Thanks!

•

dbt-bigquery==1.7.4

•

elementary-data/elementary==0.13.2

We have a hybrid dbt cloud/dbt-core architecture, and we’ve observed the

on-run-end

slowdowns on both core and cloud

According to my research with the debug option of dbt, it takes long time for elementary to deal with manifests and results with the macros. As far as I know, it is short to insert records to a warehouse.

hi @Noa Dagan, is there any updates on this? I’m encountering the same issue in snowflake, both in dbt-core and dbt-cloud. I couldn’t find any workaround in the doc either. Looking for a way to maybe switch the warehouse for the on-run-end hook if it’s hard to fix the root cause. thanks

dbt-snowflake==1.4.4
  - package: elementary-data/elementary
    version: 0.13.0

@Noa Kurman @Ofek Weiss any updates on this? its really unclear what is going on there....

following

Bit late to the party on this, but to my understanding from the dbt documentation, vars in dbt_project.yml are not jinja-rendered, so

vars:
  disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"

shouldn't really work, right?