Hello! How can I help you with Elementary Data today? If you have a specific question about Elementary Data, its features, setup, or usage, please let me know.

A message from kapa.ai

Based on the most recent information in the knowledge sources, there is not yet an official solution from Elementary Data for the issue where the on-run-end hook is triggered after every model/task when using Astronomer Cosmos, leading to performance bottlenecks—especially on systems like Redshift and Athena that struggle with concurrent transactions. The main workaround suggested is to disable the automatic artifact upload in the on-run-end hook by setting:

yaml vars:   disable_dbt_artifacts_autoupload: true

Then, you can run a separate job (e.g.,

dbt run -s elementary

) to load the artifacts in a single batch, which avoids the concurrency issue. However, for uploading run results and test results, there is no out-of-the-box mechanism to collect artifacts from S3 (or other object storage) and insert them into Elementary tables. This was recognized as a need, and the Elementary team expressed openness to further discussion, but no ready-made solution was provided in the thread or subsequent updates in the knowledge sources. If you need this functionality, the best path forward is to follow the workaround above and/or reach out to the Elementary team to discuss your use case and potential solutions, as they are aware of the challenge but have not yet released a dedicated fix or feature for it as of the latest available information[Elementary + Cosmos bottleneck discussion](https://slack.com/archives/C02CTC89LAX/p1740748388054459). If you have seen a recent Cosmos release, there is no evidence in the knowledge sources that it directly addresses this Elementary integration issue.

A message from kapa.ai

Disabling artifact upload and running it at the end of your workflow does help with metadata tables, but as you noted, run results and test results (e.g., uploading run_results.json to Elementary tables) are still append-only operations that occur on every dbt invocation. Regarding performance: - The runtime for these append-only operations (inserting run results and invocation details) generally depends on the number of models/tests executed in the run. For most users, the time added by these operations is not significant, as each run typically creates a single INSERT query for run results and another for invocation details. However, if you have a very large number of tests or models, the time can add up, especially if not chunked or optimized for your data warehouse. - In practice, users have reported that these operations can take anywhere from a few seconds to several minutes, depending on the scale. For example, one user reported that inserting 10,000 test results could take several hours if not chunked properly, but with chunking and recent bug fixes, performance improved significantly. The Elementary team fixed a bug related to chunking in version 0.14.1, which helps with large inserts by breaking them into smaller batches, thus improving performance on warehouses like Snowflake and BigQuery. Setting the chunk size (e.g.,

dbt_artifacts_chunk_size: 5000

) can help manage this further. Still, the total time is proportional to the number of rows being inserted and the performance of your data warehouse[Chunking and performance discussion](https://slack.com/archives/C02CTC89LAX/p1706626800346159). - Even with chunking, if you have a very high volume of test results, the cumulative time can be substantial. For example, one user calculated that at 3 seconds per test for 10,000 tests, the total runtime could exceed 8 hours if not optimized. In summary: Appending run results and test results is generally efficient for most workflows, but for very large runs, it can become a bottleneck. Chunking and recent improvements help, but the total time will still scale with the number of results. Running all inserts at once at the end (batching) is more efficient than many small inserts, but the impact depends on your warehouse and the volume of data. Monitoring your warehouse's query history is the best way to understand the actual impact in your environment[Run results performance details](https://slack.com/archives/C02CTC89LAX/p1679699743309069). If you need more granular control or have extreme scale, you may need to further customize your workflow or reach out to the Elementary team for advanced guidance.

A message from kapa.ai

If you are using Astronomer Cosmos and each Airflow task runs only a single dbt model, the run results per task will indeed be very small—typically just one row per model execution. In this scenario, the Elementary on-run-end hook will perform a single INSERT operation per task to log the run result and invocation details. According to the knowledge sources, when each dbt run (or task) only executes one model or test, the impact on your data warehouse is minimal: "it's gonna be a super lightweight and single insert query per run (like inserting one row)" [Airflow per-model task discussion](https://slack.com/archives/C02CTC89LAX/p1662634972514889). This means the performance overhead for each task is negligible, and you are unlikely to encounter significant delays or bottlenecks from these append-only operations, even when running many tasks in parallel. The main performance concerns and bottlenecks discussed in the knowledge sources arise when large batches of results are inserted at once, or when there are concurrency issues on certain warehouses. For your use case—one model per task—the Elementary inserts should be fast and not a limiting factor. If you do encounter issues, they are more likely to be related to warehouse-specific concurrency or transaction handling, rather than the volume of data per insert. But for the described Cosmos setup, the expected impact is very low.