@kapa.ai can you help me with this?

I'm happy to help! However, your question "can you help me with this?" is quite general, and I need more details to provide specific assistance regarding Elementary Data. If you can clarify the issue or provide more context, I'll do my best to assist you based on the available knowledge sources. If your question is about troubleshooting, configuration, or using Elementary Data features, please describe the problem or what you're trying to achieve. If you need support, you can also reach out to the Elementary team via their [Slack community](https://elementary-data.com/community) or check their documentation for more guidance. Let me know how I can assist you further!

The extremely long runtime of your Elementary

on-run-end

hook (nearly two hours) is a known issue that can occur in large dbt projects or with certain configurations. Based on your logs and configuration, here are the most relevant points and recommendations from the knowledge sources: ### Why is this happening? - Elementary’s

on-run-end

hook uploads metadata and results for all resources in your dbt project, not just those selected in the run. For large projects, this can take a significant amount of time, especially if you have thousands of models, tests, or other resources. Even with chunking and batching, the process can be slow if the project is very large or if the warehouse is under heavy load or has performance constraints [[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. - Caching and dbt version: If you are using dbt 1.4.0 or above, Elementary uses a metadata cache to avoid re-uploading unchanged artifacts. However, even with caching, the process of checking each resource can be slow for very large projects [[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. - The logs can be misleading: The time between "Running 1 on-run-end hook" and "START hook" is spent compiling and preparing the hook, which is when most of the work happens [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1701887161730629)]. ### What can you do to improve performance? #### 1. Disable automatic artifact uploads and run them manually You can disable the automatic upload of dbt artifacts on every run and instead upload them only when your project changes (e.g., after merging a PR or during deployment). This is the most effective way to reduce the

on-run-end

runtime for large projects. Add this to your `dbt_project.yml`:

yaml vars:   disable_dbt_artifacts_autoupload: true

Then, manually upload artifacts when needed:

bash dbt run --select edr.dbt_artifacts

This approach is recommended in the docs and by multiple users who have experienced similar issues [[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. #### 2. Fine-tune which hooks run and when You can use variables to disable specific parts of the

on-run-end

logic except in production, for example:

yaml vars:   disable_run_results: "{{ target.name != 'prod' }}"   disable_tests_results: "{{ target.name != 'prod' }}"   disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"

This ensures that heavy operations only run in your production environment [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1706760230231499)]. #### 3. Project size and warehouse performance The size of your dbt project (number of models, tests, etc.) and the performance of your data warehouse are major factors. If possible, review whether all resources need to be included in Elementary monitoring, or if you can reduce the scope. #### 4. Check for dynamic expressions If you have dynamic expressions in your model configs (e.g., using

now()

or

var()

), this can cause the cache to be invalidated on every run, leading to full re-uploads each time [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1680096134347739)]. --- Summary: Your configuration is already using chunking, but for very large projects, the best way to reduce

on-run-end

time is to disable automatic artifact uploads and run them manually only when needed. This is a common solution for teams with large dbt projects and is well-supported by the Elementary documentation and community experience [[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. If you need more tailored advice, sharing your

dbt.log

with the Elementary team can help them diagnose any additional bottlenecks.

A message from kapa.ai

The difference between dbt artifacts and runtime metadata in the context of Elementary is as follows: - dbt artifacts refer to the metadata about your dbt project—this includes information about models, tests, sources, exposures, and their configurations. These are typically static details that describe the structure and setup of your dbt project. In Elementary, these are stored in tables like

dbt_models

,

dbt_tests

,

dbt_sources

, etc. Uploading these artifacts ensures that Elementary has the latest view of your project’s structure and configuration [[Elementary dbt artifacts](https://docs.elementary-data.com/dbt/dbt-artifacts)]. - Runtime metadata (or "result models") refers to the logs and results generated when you actually run dbt commands. This includes run results, test results, and invocation details—such as model run times, statuses, errors, and other execution metrics. These are stored in tables like

dbt_run_results

,

elementary_test_results

, and

dbt_invocations

[[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. If you add

disable_dbt_artifacts_autoupload: true

to your config, it will only disable the automatic upload of the project metadata (dbt artifacts) on every run. **It will not limit your ability to track runtime metrics**—the run results and test results will still be uploaded unless you also set the following variables:

yaml vars:   disable_run_results: true   disable_tests_results: true   disable_dbt_invocation_autoupload: true

So, with only

disable_dbt_artifacts_autoupload: true

, you will still be able to track runtime metrics and see execution results in Elementary. You just need to remember to manually update the project metadata (artifacts) when your dbt project changes, for example by running:

bash dbt run --select edr.dbt_artifacts

[[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks#can-i-disable-the-on-run-end-hooks%3F)] In summary: - Disabling artifact autoupload does not stop runtime metrics from being tracked. - It only stops automatic updates of project metadata; runtime results are still collected unless you disable those separately.

The difference between dbt artifacts and runtime metadata in the context of Elementary is as follows: - dbt artifacts refer to the metadata about your dbt project—this includes information about models, tests, sources, exposures, and their configurations. These are typically static details that describe the structure and setup of your dbt project. In Elementary, these are stored in tables like

dbt_models

,

dbt_tests

,

dbt_sources

, etc. Uploading these artifacts ensures that Elementary has the latest view of your project’s structure and configuration [[Elementary dbt artifacts](https://docs.elementary-data.com/dbt/dbt-artifacts)]. - Runtime metadata (or "result models") refers to the logs and results generated by each dbt run—such as run times, statuses, errors, and test outcomes. These are dynamic and change with each invocation. In Elementary, these are stored in tables like

dbt_run_results

,

elementary_test_results

, and

dbt_invocations

[[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. If you add

disable_dbt_artifacts_autoupload: true

to your config, it will only disable the automatic upload of the project metadata (dbt artifacts) on each run. **It will not limit your ability to track runtime metrics**—the run results and test results will still be uploaded unless you also set the following variables:

yaml vars:   disable_run_results: true   disable_tests_results: true   disable_dbt_invocation_autoupload: true

So, with only

disable_dbt_artifacts_autoupload: true

, you will still be able to track runtime metrics such as model run times, test outcomes, and invocation details. You just need to remember to manually update the project metadata (artifacts) when your dbt project changes, for example by running:

bash dbt run --select edr.dbt_artifacts

[[Elementary dbt on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks#can-i-disable-the-on-run-end-hooks%3F)] In summary: Disabling artifact autoupload does not affect runtime metrics tracking; it only affects the automatic update of project metadata. Runtime metrics will continue to be tracked unless you explicitly disable them with the other variables.

A message from kapa.ai

The difference between dbt artifacts and runtime metadata in Elementary is: - dbt artifacts are the metadata about your dbt project—such as models, tests, sources, exposures, and their configurations. These describe the structure and setup of your project and are stored in tables like

dbt_models

,

dbt_tests

,

dbt_sources

, etc. This metadata is updated when you run the relevant Elementary models or via the

on-run-end

hook, and it only represents the current state of your project [[Elementary dbt artifacts](https://docs.elementary-data.com/dbt/dbt-artifacts)][[Elementary on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. - Runtime metadata (also called "result models") refers to the logs and results generated by each dbt run—such as run results, test results, and invocation details. These are dynamic and change with each invocation, and are stored in tables like

dbt_run_results

,

elementary_test_results

, and

dbt_invocations

[[Elementary on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks)]. If you add

disable_dbt_artifacts_autoupload: true

to your config, it will only disable the automatic upload of project metadata (dbt artifacts) on every run. **It will not limit your ability to track runtime metrics**—run results and test results will still be uploaded unless you also set the following variables:

yaml vars:   disable_run_results: true   disable_tests_results: true   disable_dbt_invocation_autoupload: true

So, with only

disable_dbt_artifacts_autoupload: true

, you will still be able to track runtime metrics such as model run times and test outcomes. You just need to manually update the project metadata (artifacts) when your dbt project changes, for example by running:

bash dbt run --select edr.dbt_artifacts

[[Elementary on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks#can-i-disable-the-on-run-end-hooks%3F)] Summary: Disabling artifact autoupload does not affect runtime metrics tracking; it only affects the automatic update of project metadata. Runtime metrics will continue to be tracked unless you explicitly disable them with the other variables.