Yes, you can improve the performance and reduce the number of queries related to the

dbt_run_results

table in Elementary by tuning several configuration options and understanding how the on-run-end hooks work. ### Key Strategies #### 1. Upgrade to dbt 1.4.0 or Above Elementary leverages a metadata cache in dbt 1.4.0+, which significantly reduces unnecessary updates and queries for metadata models. If you are on an older version, every run will update all metadata, increasing query load. Upgrading is strongly recommended for better performance [Elementary on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks). #### 2. Disable Automatic Artifact Uploads If you don't need to update metadata on every run, you can disable the automatic upload of dbt artifacts. Add the following to your `dbt_project.yml`:

yaml vars:   disable_dbt_artifacts_autoupload: true

Then, manually run

dbt run --select edr.dbt_artifacts

when you want to update the metadata (e.g., after merging a PR). This reduces the number of queries on regular runs [Elementary on-run-end hooks](https://docs.elementary-data.com/dbt/on-run-end_hooks). #### 3. Control Run Results and Test Results Uploads You can further reduce query volume by disabling run results and test results uploads if they are not needed for your use case:

yaml vars:   disable_run_results: true   disable_tests_results: true   disable_dbt_invocation_autoupload: true

This is especially useful if you only want to collect results periodically or in specific environments [Slack thread](https://slack.com/archives/C02CTC89LAX/p1673378114992729). #### 4. Reduce Batch Size for Artifact Uploads If you are hitting query size limits (especially on BigQuery), reduce the batch size for artifact uploads:

yaml vars:   dbt_artifacts_chunk_size: 25

You may need to experiment with this value; some users have gone as low as 100 or even 25 to avoid query length errors [GitHub Issue](https://github.com/elementary-data/elementary/issues/1689), [Slack thread](https://slack.com/archives/C02CTC89LAX/p1663773279382749). #### 5. Optimize Query Size If you encounter errors like "The query is too large," you can set a lower `query_max_size`:

yaml vars:   query_max_size: 250000

Adjust this value as needed to stay under your data warehouse's query length limits [Slack thread](https://slack.com/archives/C02CTC89LAX/p1678194978401139). ### Additional Notes - The number of rows inserted into

dbt_run_results

depends on the number of models/tests executed in the run. Large projects or frequent runs will naturally generate more queries. - If you disable some uploads, ensure you still update the necessary tables when you need accurate reports or alerts. - For Databricks and other warehouses, query execution time can also be affected by warehouse performance, not just Elementary configuration [Slack thread](https://slack.com/archives/C02CTC89LAX/p1705651386792909). By applying these strategies, you can significantly reduce the number of queries and improve the performance of Elementary's integration with your dbt runs.

The behavior you're seeing—where the incremental model

dbt_run_results

processes a large amount of data (e.g., "24.1 GiB processed") on every PR run—can occur if the incremental logic is not functioning as expected, or if the table is being rebuilt or fully scanned each time. From the knowledge sources, here are some key points to consider: - Elementary's

dbt_run_results

model is incremental by default, and should only process new data unless something is causing a full scan or rebuild. If you see large data volumes processed on every run, it may be due to a misconfiguration or an override in your

dbt_project.yml

that forces a full refresh or disables incremental behavior. It's also possible that a full-refresh is being triggered (either explicitly or by CI defaults) which would cause the table to be rebuilt and all data to be processed again [Slack thread](https://slack.com/archives/C02CTC89LAX/p1668519863054579). - Full-refresh runs clear incremental tables: If your PR jobs are running with

--full-refresh

, this will cause all incremental models (including

dbt_run_results

) to be rebuilt from scratch, resulting in high data processing volumes. The recommendation in such cases is to exclude Elementary models from full-refresh runs if you want to preserve incremental behavior:

dbt run --exclude elementary --full-refresh

[Slack thread](https://slack.com/archives/C02CTC89LAX/p1668519863054579) - Check for overrides in your configuration: If there are any materialization or variable overrides in your

dbt_project.yml

or in your CI/CD pipeline that affect Elementary models, these could be causing the incremental logic to be bypassed. For example, setting

disable_run_results

or similar variables to

True

can affect what gets logged and how the models behave [Slack thread](https://slack.com/archives/C02CTC89LAX/p1668519863054579). - Incremental logic depends on the warehouse and dbt version: In rare cases, mismatches between dbt versions or warehouse adapters can cause incremental models to not function as expected. Ensuring your dbt and adapter versions are compatible with Elementary is important [Slack thread](https://slack.com/archives/C02CTC89LAX/p1673534189748399). Summary: If your PR runs are processing large amounts of data for incremental Elementary models, check if a full-refresh is being triggered, or if there are configuration overrides affecting incremental behavior. Exclude Elementary from full-refresh runs if you want to retain incremental processing, and review your

dbt_project.yml

and CI/CD settings for any relevant overrides. If you need to retain history but avoid heavy processing on every PR, consider copying the results to your own tables after each run, as some users have done [Slack thread](https://slack.com/archives/C02CTC89LAX/p1668519863054579).

A message from kapa.ai

Based on the knowledge sources, your

vars

configuration:

yaml vars:   'dbt_date:time_zone': 'Europe/London'   disable_dbt_artifacts_autoupload: true

is valid and should not, by itself, cause the incremental model

elementary.dbt_run_results

to process large amounts of data on every PR run. Setting

disable_dbt_artifacts_autoupload: true

is the recommended way to disable automatic artifact uploads and should reduce unnecessary processing if you are not running full-refresh or overriding materializations elsewhere. However, if you are still seeing large data volumes processed, the issue is likely not with your

vars

section but could be related to: - Running dbt with the

--full-refresh

flag, which rebuilds incremental models from scratch and causes all data to be processed, regardless of the

vars

settings. - Other configuration or CI/CD pipeline settings that might override the incremental logic or trigger full-refresh behavior. If you are not running with

--full-refresh

and do not have other overrides, your

vars

configuration appears correct according to the knowledge sources. If the problem persists, further investigation into your dbt run commands and pipeline configuration would be necessary, as the knowledge sources do not provide additional details for this specific scenario [Slack thread](https://slack.com/archives/C02CTC89LAX/p1668519863054579).

To stop large

dbt_run_results

processing on your PR runs when using:

dbt build --select state:modified+

Key points from the knowledge sources: - The

dbt_run_results

model in Elementary is incremental by default and should only process new data. However, if it is still processing large amounts of data, it may be because the model is being selected and run even when not necessary, or because of how dbt selectors work with Elementary models. - When using selectors like

state:modified+

, if Elementary models are included in the selection, they will run and process data. This can be especially heavy if your project is large or if the artifacts are being rebuilt or merged unnecessarily. - One recommended approach is to exclude Elementary models from your PR runs unless you specifically need to update them. You can do this by modifying your command to:

dbt build --select state:modified+ --exclude elementary

This will prevent Elementary models, including

dbt_run_results

, from running during PR jobs, thus avoiding the large data processing. - If you need to run Elementary models only in certain environments (e.g., production), you can also control this via your `dbt_project.yml`:

yaml   models:     elementary:       +enabled: "{{ target.name == 'prod' }}"

This ensures Elementary models are only built in production, not in PR or dev environments [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1672228916705469). Summary: To stop large

dbt_run_results

processing on PRs, exclude Elementary models from your PR build command using

--exclude elementary

. This is the most direct and supported way to avoid unnecessary processing according to the knowledge sources.