Thank you for the detailed context. Here’s what the knowledge sources say about running Elementary in parallelized Airflow environments (like Cosmos), and how to manage concurrency issues with Elementary’s auto-generated models: --- ### 1. Recommended Approach to Prevent Concurrent Modifications Concurrency issues (such as “Transaction is aborted due to concurrent update against table dbt_tests”) are a known problem when multiple dbt runs (or Airflow tasks) attempt to write to the same Elementary-managed tables at the same time. This is especially common when using orchestrators like Airflow with Cosmos, which parallelize dbt model execution. Recommended Workaround: - Disable automatic artifact uploads on every dbt run by setting the following in your `dbt_project.yml`:

yaml   vars:     disable_dbt_artifacts_autoupload: true

- Instead, run

dbt run -s elementary

(or

dbt run --select elementary

) in a single, dedicated job after your parallelized model runs are complete. This ensures that only one process is updating the Elementary artifact tables at a time, avoiding concurrent writes and transaction conflicts. This approach is explicitly recommended by Elementary support for Airflow/Cosmos and similar orchestrated environments to avoid concurrency issues with metadata tables like

dbt_models

,

dbt_tests

, etc. [See: Slack discussion on Cosmos and concurrency](https://slack.com/archives/C02CTC89LAX/p1740748388054459), [Another Slack thread on disabling autoupload and running a single job](https://slack.com/archives/C02CTC89LAX/p1671417541688279). --- ### 2. Tagging or Grouping Elementary Models for Isolation Elementary’s models (like

dbt_tests

,

dbt_models

, etc.) are grouped under the

elementary

selector. This means you can always refer to them collectively using

-s elementary

or

--select elementary

in dbt commands. - There is no evidence in the knowledge sources that Elementary provides custom tags or selectors beyond this. - The recommended practice is to use the built-in

elementary

selector to run all Elementary models together, in a controlled, non-parallelized step. This makes it easy to isolate Elementary’s models from your main transformation models and ensures you can control when and how they are run [See: Cosmos/Airflow guidance](https://slack.com/archives/C02CTC89LAX/p1740748388054459). --- ### 3. Official Guidance for Cosmos and Dynamic Parallelization - Elementary’s official guidance for orchestrated environments (like Airflow with Cosmos) is to avoid running Elementary models in parallel with your main dbt models. - Instead, run all your transformation models in parallel as needed, but then run a single, serial job for the Elementary models using

dbt run -s elementary

after all parallel tasks are complete. This is the only approach currently recommended by Elementary support to avoid transaction conflicts and ensure reliable artifact collection [See: Cosmos/Airflow Slack thread](https://slack.com/archives/C02CTC89LAX/p1740748388054459), [General concurrency guidance](https://slack.com/archives/C02CTC89LAX/p1671417541688279). --- #### Summary Table | Problem | Solution | |------------------------|------------------------------------------------------------------------------------------| | Concurrent writes to Elementary tables | Disable auto-upload (

disable_dbt_artifacts_autoupload: true

), run

dbt run -s elementary

in a single job | | Isolating Elementary models | Use the built-in

elementary

selector (

-s elementary

) to group and run them together | | Cosmos/Airflow integration | Do not parallelize Elementary models; run them in a dedicated, serial step after all parallel model runs | --- If you need to further batch or aggregate results (e.g., from S3 or object storage), this is not natively supported by Elementary as of now, but has been discussed as a possible future enhancement [See: Cosmos/Airflow Slack thread](https://slack.com/archives/C02CTC89LAX/p1740748388054459). If you have more specific needs (like custom artifact aggregation), the Elementary team is open to discussion for advanced use cases. --- References: - [Cosmos/Airflow concurrency and Elementary guidance](https://slack.com/archives/C02CTC89LAX/p1740748388054459) - [Disabling autoupload and running a single job](https://slack.com/archives/C02CTC89LAX/p1671417541688279) If you need more details or want to discuss advanced integration, the Elementary team encourages reaching out for a deeper conversation.