hey Team, after upgrading to 0.14.1 we have an issue: on-run-end hook is very slow: it takes up to 3 mins vs 10 seconds on version 0.11.2. Is there a way to understand what slows it down?

Hi @Oleksii Nikitchuk! sorry to hear about it, you can look at the dbt.log and see which part is taking the longest. It might be because we now also upload metadata about columns. You can try to optimize it by configuring this --> You can lower the number of columns metadata that are being uploaded

vars:
  upload_only_columns_with_descriptions: true

Or disable it altogether -->

vars:
  disable_dbt_columns_autoupload: true

More context and suggestions --> On the

on-run-end

hooks Elementary extracts data from the dbt

results

and

metadata

objects, and runs SQL queries to load this data to the Elementary models. There are 2 types of models that Elementary updates : 1. Metadata and project objects and configurations - Such as

dbt_models

,

dbt_tests

,

dbt_sources

. 2. Result models - Such as

dbt_run_results

,

elementary_test_results

,

dbt_invocations

. 1. So you can disable the auto uploading of the metadata models (results have to be uploaded during the on-run-end). 2. Make sure to upload the metadata models yourself anytime you make a change to the project (merge a PR). How? 1. Add this to dbt_project.yml:

vars:
  disable_dbt_artifacts_autoupload: true

2. Make sure to run

dbt run --select edr.dbt_artifacts

upon merging PRs. you can also read more about it here --> https://docs.elementary-data.com/dbt/on-run-end_hooks#performance-impact-of-on-run-end-hooks

thanks @Or Avidov! will try it