https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> transaction aborted due to concurre...
# ask-elementary-ai
s
@kapa.ai transaction aborted due to concurrent update against table
k
This error is a known concurrency issue when Elementary’s dbt models or hooks try to update the same Elementary-managed table at the same time (for example 
dbt_models
, 
dbt_tests
, or other 
dbt_artifacts
tables), especially on BigQuery and in orchestrated environments like Airflow/dbt Cloud. [BQ concurrency; Concurrent DBT jobs] Typical message (BigQuery example): > Transaction is aborted due to concurrent update against table project`:elementary.dbt_models` This happens when multiple dbt runs (or DAGs) execute Elementary’s on-run-end hooks or 
dbt_artifacts
models concurrently, so two transactions try to mutate the same table and BigQuery (or another warehouse) aborts one. [BQ concurrency; Issue 1766] Mitigations / workarounds from the knowledge sources 1. Avoid concurrent Elementary writes (recommended pattern) ◦ Disable automatic dbt artifacts upload in hooks:
Copy code
vars:
         disable_dbt_artifacts_autoupload: true
[BQ concurrent runs] ◦ Then run a single, dedicated job after your main pipelines:
Copy code
dbt run -s elementary
or
Copy code
dbt run --select elementary
This centralizes all Elementary artifact updates into one non-parallel step, preventing concurrent updates to tables like 
dbt_models
/ 
dbt_tests
. [Airflow concurrency thread; BQ concurrent runs] 2. Serialize dbt jobs that run Elementary hooks If you must keep hooks enabled, use your orchestrator to ensure only one job that runs Elementary hooks is active at a time (e.g., Airflow pool with 1 slot for dbt+Elementary tasks). This has been confirmed to stop the error in practice. [BQ concurrency thread; Airflow/BQ error] 3. dbt Cloud / general concurrent jobs For dbt Cloud or any environment with many concurrent dbt jobs, this is acknowledged as a known limitation; the recommended direction is again to avoid parallel writes to Elementary tables (via the patterns above). [dbt Cloud concurrency] If you share your warehouse (BigQuery, Databricks, Spark, etc.) and how you orchestrate dbt (Airflow, dbt Cloud, Cosmos), I can point you to the most relevant of these patterns. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai Warehouse BigQuery, orchestration through Airflow
k
For BigQuery + Airflow, the “transaction is aborted due to concurrent update against table …elementary.dbt_models/dbt_tests/etc.” error is a known Elementary concurrency issue when multiple dbt runs (DAGs/tasks) update the same Elementary tables at the same time. [BQ concurrency; Concurrent runs workaround] Here are the patterns that Elementary support explicitly recommends for BigQuery + Airflow: ------------------------- 1. Disable artifact autoupload in hooks (recommended) In `dbt_project.yml`:
Copy code
vars:
  disable_dbt_artifacts_autoupload: true
This stops the on-run-end hook from mutating the heavy 
dbt_artifacts
tables (like 
dbt_models
, 
dbt_sources
, 
dbt_tests
) on every dbt run, which is where most BigQuery transaction conflicts occur. [Disable autoupload] Then, run artifacts in a single, non-parallel job (e.g. a dedicated Airflow task/DAG):
Copy code
dbt run --select elementary.edr.dbt_artifacts
# or simply
dbt run -s elementary
Do this after your main dbt DAGs finish, or on deploy / on a schedule. This centralizes all artifact updates into one transaction stream and removes concurrent updates. [Disable autoupload workaround; Airflow hook discussion] Note: this only disables artifacts upload; test results and invocation uploads are controlled by other vars and can remain enabled. [Artifacts vs invocations] ------------------------- 2. Serialize dbt+Elementary jobs via Airflow pools If you keep hooks enabled, you can avoid conflicts by ensuring only one dbt run that executes Elementary hooks is active at a time: • Put all dbt tasks that run with Elementary hooks into an Airflow pool with 
slots=1
. • Other non-Elementary tasks can use different pools. Users have confirmed that serializing these jobs via an Airflow pool removes the BigQuery “transaction aborted due to concurrent update” errors. [BQ concurrency + Airflow pool] ------------------------- 3. Cosmos / per-model tasks (if applicable) If you’re using Astronomer Cosmos to map each dbt model to an Airflow task, Elementary strongly recommends: • Do not let every per-model task run the Elementary on-run-end hook. • Instead: ◦ Disable artifact autoupload as above. ◦ Add a single downstream task/DAG that runs:
Copy code
dbt run -s elementary
after all parallel model tasks complete. [Cosmos concurrency guidance; Airflow/Cosmos summary] ------------------------- 4. What this means in practice for your setup For BigQuery + Airflow, the safest and most common pattern from the sources is: 1. Set in `dbt_project.yml`:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
2. In Airflow: ◦ Let your normal dbt DAGs (models/tests) run in parallel as needed. ◦ Add a separate DAG or final task that runs:
Copy code
dbt run -s elementary
using a pool or configuration that ensures it runs alone (no other dbt+Elementary jobs at the same time). This combination is what has resolved the BigQuery “transaction is aborted due to concurrent update” errors for other users in similar environments. [BQ concurrent runs; Airflow + BQ error] If you share a bit about how your Airflow DAGs are structured (single big dbt run vs many small runs / Cosmos), I can suggest a concrete DAG pattern based on these recommendations. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai what more information would I need ?
k
To move from generic guidance to a concrete fix for your setup, the knowledge sources suggest you’d want to clarify at least: 1. How dbt is invoked from Airflow ◦ Single “big” 
dbt run
per DAG, or many small runs? ◦ Are you using Cosmos (
DbtTaskGroup
/ 
DbtLSOperator
) so that each model is its own Airflow task? [Cosmos concurrency] 2. Where Elementary hooks are running ◦ Are Elementary’s 
on-run-end
hooks enabled on every dbt command? ◦ Have you already set:
Copy code
vars:
         disable_dbt_artifacts_autoupload: true
in 
dbt_project.yml
, or not yet? [Disable autoupload] 3. Concurrency pattern in Airflow ◦ How many dbt tasks can run in parallel (DAG-level concurrency, pools, queues)? ◦ Are multiple DAGs with dbt+Elementary running at the same time? [BQ concurrency + pools] 4. Which Elementary tables are failing ◦ Exact error text and table name: 
dbt_models
, 
dbt_tests
, 
dbt_sources
, or 
dbt_run_results
? This matters because: ▪︎ 
dbt_artifacts
tables (e.g. 
dbt_models
) are the main source of conflicts and are controlled by 
disable_dbt_artifacts_autoupload
. [Artifacts concurrency] ▪︎ Result tables like 
dbt_run_results
/ 
elementary_test_results
are append-only and trickier to batch. [Cosmos batching discussion] 5. Whether you already run a separate Elementary job ◦ Do you currently have a dedicated task/DAG that runs:
Copy code
dbt run -s elementary
after your main dbt runs, or is everything happening only via hooks? [Single artifacts job] With those details, we could map you to one of the documented patterns (e.g., “disable autoupload + single 
dbt run -s elementary
DAG” vs. “serialize dbt+Elementary tasks via an Airflow pool”) in a very concrete way. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai to answer that I have not set up
Copy code
vars:
         disable_dbt_artifacts_autoupload: true
I have only made it that "dbt run --select elementary" runs when ever a new docker image is made
k
You’re very close to the recommended pattern, but with one important difference. From the knowledge sources, the main cause of your BigQuery “transaction aborted due to concurrent update” errors is the on‑run‑end hook updating 
dbt_artifacts
tables (e.g. 
dbt_models
, 
dbt_tests
, 
dbt_sources
) while multiple dbt runs are executing in parallel. [BQ concurrency; Artifacts concurrency workaround] Running:
Copy code
dbt run --select elementary
only on new Docker images (i.e., on deploy) is good, but it does not stop the hooks from still trying to update those same tables on every dbt run. To actually remove the conflicting writes, the docs and Slack threads say you should: 1. Disable artifact autoupload in hooks (this is the missing piece in your setup):
Copy code
# dbt_project.yml
   vars:
     disable_dbt_artifacts_autoupload: true
This tells Elementary’s on‑run‑end hook to skip updating 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc. [On‑run‑end docs; BQ concurrent runs] 2. Keep your separate 
dbt run --select elementary
job What you’re already doing (running 
dbt run --select elementary
on new images) is exactly what Elementary recommends once autoupload is disabled: > “After disabling the automatic upload, you can run your artifacts update process independently… remember to run 
dbt run --select elementary.edr.dbt_artifacts
whenever you deploy changes to your project…” [BQ concurrent runs] You can keep it as:
Copy code
dbt run --select elementary
or narrow it to:
Copy code
dbt run --select elementary.edr.dbt_artifacts
So, what you still need to do: • Add:
Copy code
vars:
      disable_dbt_artifacts_autoupload: true
to 
dbt_project.yml
. • Keep your existing “on new Docker image” job that runs 
dbt run --select elementary
(or 
elementary.edr.dbt_artifacts
). That combination matches the documented workaround for BigQuery + Airflow concurrency and should significantly reduce or eliminate the “transaction aborted due to concurrent update” errors on the 
dbt_artifacts
tables. [BQ concurrency workaround; Airflow concurrency thread] If you later still see errors, the next lever (from the sources) would be to serialize dbt+Elementary runs via an Airflow pool, but usually the 
disable_dbt_artifacts_autoupload
change is enough for BigQuery. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai - name: Run DBT Elementary Init # id: run-dbt-elementary-init-command # if: needs.find_environments_and_files_to_deploy.outputs.rebuild_image == 'true' # run: | # curl --fail-with-body -d '{"command": "run --select elementary"}' \ # -H "Authorization: Bearer ${{ steps.auth-full.outputs.id_token }}" \ # -H "Content-Type: application/json" \ # -X POST "${{ env.cloudrun_http }}/"
k
You’re already very close to the recommended pattern. Based on the knowledge sources, here’s how to align this snippet with Elementary’s guidance for avoiding BigQuery concurrency issues: 1. Keep a dedicated “init” run for Elementary models Having a separate job that runs:
Copy code
dbt run --select elementary
only when you rebuild the image / deploy is exactly what Elementary suggests for loading artifacts in a single, controlled job. [BQ concurrent runs] Your (commented) step:
Copy code
- name: Run DBT Elementary Init
     # id: run-dbt-elementary-init-command
     # if: needs.find_environments_and_files_to_deploy.outputs.rebuild_image == 'true'
     run: |
       curl --fail-with-body -d '{"command": "run --select elementary"}' \
           -H "Authorization: Bearer ${{ steps.auth-full.outputs.id_token }}" \
           -H "Content-Type: application/json" \
           -X POST "${{ env.cloudrun_http }}/"
is consistent with that idea: you’re triggering a dbt command that runs all 
elementary
models once per deploy. 2. What’s still missing: disable artifact autoupload in hooks To actually prevent the “transaction aborted due to concurrent update” errors, the sources say you must also disable automatic artifact uploads in the on‑run‑end hook:
Copy code
# dbt_project.yml
   vars:
     disable_dbt_artifacts_autoupload: true
This stops every dbt run from updating 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc., and leaves your dedicated 
run --select elementary
job as the only process that updates those tables. [Disable autoupload; Airflow/concurrency guidance] 3. Putting it together ◦ In `dbt_project.yml`:
Copy code
vars:
         disable_dbt_artifacts_autoupload: true
◦ In CI/CD (your snippet), keep or re‑enable the “Run DBT Elementary Init” step so that on image rebuild / deploy you call:
Copy code
dbt run --select elementary
That combination matches the documented workaround: artifacts are no longer written on every run (avoiding concurrent updates), and are instead refreshed in a single, explicit job. [BQ concurrent runs] If you share how this Cloud Run endpoint executes dbt (e.g., which profile/target), we could double‑check that it’s hitting the same project/warehouse as your Airflow jobs, but the pattern itself is correct per the sources. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai this is in the build-deploy-datalake.yml
k
Given the knowledge sources, the missing piece is not where this step lives (in 
build-deploy-datalake.yml
), but that you still need to disable Elementary’s artifact autoupload in 
dbt_project.yml
 so that this deploy-time run becomes the only place artifacts are written. From the docs and Slack threads, the recommended pattern for avoiding BigQuery “transaction aborted due to concurrent update” is: 1. In 
dbt_project.yml
, add:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
This stops the on‑run‑end hook from updating 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc. on every dbt run. [On‑run‑end docs; BQ concurrency workaround] 2. Keep your deploy-time job in 
build-deploy-datalake.yml
that triggers:
Copy code
dbt run --select elementary
(or more narrowly 
dbt run --select elementary.edr.dbt_artifacts
) whenever you build/deploy the image. This is exactly what Elementary recommends once autoupload is disabled: run the artifacts models in a single, controlled job on deploy. [BQ concurrency workaround; Airflow concurrency thread] So your 
build-deploy-datalake.yml
step is in the right place; the key change is to pair it with 
disable_dbt_artifacts_autoupload: true
in 
dbt_project.yml
so that Airflow runs no longer try to update the same Elementary tables concurrently. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai does the artifacts models have to be in the dbt run --select elementary ?
k
No, they don’t have to be, but it’s usually what you want. From the docs: • To update all Elementary models you can run:
Copy code
dbt run --select elementary
• If you only want to refresh the artifacts (metadata) tables (like 
dbt_models
, 
dbt_tests
, 
dbt_sources
), you can run just the artifacts sub‑selector:
Copy code
dbt run --select elementary.edr.dbt_artifacts
This is what Elementary explicitly recommends when you disable autoupload with 
disable_dbt_artifacts_autoupload: true
and want to update artifacts manually on deploys. [On‑run‑end hooks; Slack manual artifacts run] So: • If your goal is just to keep project metadata up to date and avoid concurrency issues, you can safely run only 
elementary.edr.dbt_artifacts
. • If you also want other Elementary models (e.g. metadata_store, result processing models) to run, then keep 
dbt run --select elementary
. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai right now the dbt run --select elementary runs only when the Docker image is rebuilt — which is triggered by changes to packages.yml, Dockerfile, dbt_project.yml, profiles.yml, or deploy scripts merging to main.
k
You’re aligned with the recommended pattern, with one nuance to be aware of. From the sources, the key requirements when you disable artifact autoupload are: 1. Set in `dbt_project.yml`:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
This stops on‑run‑end from updating 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc. on every run. [On‑run‑end hooks; Control on‑run‑end time] 2. Refresh artifacts whenever your project structure changes (new/changed models, tests, sources, etc.), typically via:
Copy code
dbt run --select elementary.edr.dbt_artifacts
or
Copy code
dbt run --select elementary
[Control on‑run‑end time; Slack manual artifacts run] Your current setup: • 
dbt run --select elementary
runs only when the Docker image is rebuilt. • The image rebuild is triggered by changes to 
packages.yml
, 
Dockerfile
, 
dbt_project.yml
, 
profiles.yml
, or deploy scripts. This is generally acceptable, because: • Changes to 
dbt_project.yml
or 
packages.yml
usually coincide with project metadata changes, which is exactly when artifacts should be refreshed. [dbt artifacts] • Elementary only needs the artifacts models run: ◦ On initial install. ◦ On minor package upgrades. ◦ When project structure/config changes. [dbt artifacts] The only caveat (not explicitly covered but implied by the docs) is: • If you change only SQL model files or test YAMLs in a way that affects project metadata, and that does not trigger a Docker rebuild in your process, then artifacts would not be refreshed until the next rebuild. The docs say artifacts “only need to be uploaded when your project structure changes,” but they don’t prescribe exactly how you detect that. [Control on‑run‑end time] So, based on the sources: • Your pattern (refreshing Elementary on image rebuild) is consistent with the guidance, as long as image rebuilds roughly coincide with schema/config changes. • To be stricter, you could: ◦ Narrow the command to artifacts only:
Copy code
dbt run --select elementary.edr.dbt_artifacts
[Slack manual artifacts run] ◦ Or ensure that any change that affects dbt project structure also triggers a rebuild (so artifacts are refreshed promptly). Functionally, you don’t need to run 
--select elementary
more often than that for concurrency reasons; the important part is pairing this deploy-time run with 
disable_dbt_artifacts_autoupload: true
so regular Airflow runs don’t write to the artifacts tables. [Control on‑run‑end time] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai can the dbt run --select elementary.edr.dbt_artifacts be done with Airflow DAG?
k
Yes, you can run 
dbt run --select elementary.edr.dbt_artifacts
from an Airflow DAG. The knowledge sources explicitly recommend disabling artifact autoupload and then running the artifacts models from a separate job (or DAG) at a time you control, including in Airflow: • Set in `dbt_project.yml`:
Copy code
vars:
      disable_dbt_artifacts_autoupload: true
• Then, in a dedicated job (which can be an Airflow DAG/task), run:
Copy code
dbt run --select elementary.edr.dbt_artifacts
This pattern is suggested specifically to avoid concurrency issues and to “run your artifacts update process independently whenever you determine is necessary,” which includes orchestrators like Airflow. [Disable autoupload workaround; Manual artifacts run] So yes: it’s both supported and recommended to schedule that 
dbt run --select elementary.edr.dbt_artifacts
command as an Airflow DAG/task, as your central place to refresh Elementary artifacts. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai what happens if I don't update the artifacts ?
k
If you don’t update the Elementary dbt artifacts, several things degrade or break because Elementary relies on those tables to understand your current dbt project structure. From the docs and Slack threads: 1. Project metadata becomes stale or wrong Artifacts tables like 
dbt_models
, 
dbt_tests
, 
dbt_sources
, 
dbt_exposures
, etc. describe what models/tests/sources exist and how they’re configured. If you don’t rerun the artifacts models after changes, these tables will no longer match your real project. [dbt artifacts] 2. Reports and alerts may miss or misrepresent tests/models Elementary’s reports and alerts are built by joining runtime results (e.g. 
elementary_test_results
) with metadata from artifacts tables (e.g. 
dbt_tests
, 
dbt_models
). If 
dbt_tests
or related tables are empty or outdated, tests may not appear correctly in reports or invocation results. [dbt artifacts; dbt_tests empty discussion] 3. New or changed resources won’t be visible to Elementary ◦ New models/tests/sources won’t show up in 
dbt_models
/ 
dbt_tests
/ 
dbt_sources
. ◦ Config changes (severity, tags, owners, etc.) won’t be reflected. Elementary only replaces these metadata tables with the current graph when the artifacts models run. [dbt artifacts] 4. Some logic that depends on “latest project state” will be wrong The docs emphasize that metadata models “only represent the project state on the latest run.” If you stop updating them, anything that assumes they reflect the latest state (e.g., coverage views, some incident context) will be based on an old snapshot. [on‑run‑end hooks] 5. In extreme cases, tables can be empty after schema refreshes If your pipeline drops/cleans the schema containing Elementary tables and you don’t rerun 
dbt run --select elementary
(or at least 
elementary.edr.dbt_artifacts
), tables like 
dbt_tests
can end up empty, and then tests won’t appear at all in results or reports. [dbt_tests empty discussion] What does still work without artifacts updates: • Runtime tables like 
dbt_run_results
, 
elementary_test_results
, 
dbt_invocations
can still be populated by the on‑run‑end hooks, even if you disable artifact autoupload; you only lose the automatic project metadata refresh, not the run results themselves. [on‑run‑end hooks] In practice, the guidance is: • It’s fine not to update artifacts on every run, but you should update them whenever your project structure/config changes (e.g., on deploy) using something like:
Copy code
dbt run --select elementary.edr.dbt_artifacts
especially if you’ve set 
disable_dbt_artifacts_autoupload: true
. [on‑run‑end hooks; manual artifacts run] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai does the dbt run --select elementary.edr.dbt_artifacts require the CLI to be installed ?
k
No, 
dbt run --select elementary.edr.dbt_artifacts
does not require the Elementary CLI to be installed. That command is a dbt command that runs models from the Elementary dbt package inside your dbt project. It only requires: • The Elementary dbt package to be installed in your project. • A working dbt environment (dbt Core or dbt Cloud) with a valid profile. The Elementary CLI (
edr ...
) is a separate Python package used for reports, monitoring, and alerts, and is not needed to run 
dbt run
or the 
elementary.edr.dbt_artifacts
models. [Generate report guide; Elementary in production] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai how does the 
dbt run --select elementary.edr.dbt_artifacts
prevent the concurrent transaction error?
k
It doesn’t by itself prevent concurrent transaction errors—the prevention comes from the combination of: 1. Disabling automatic artifact uploads in the on‑run‑end hook, and 2. Running the artifacts models once in a single, controlled job, e.g. via:
Copy code
dbt run --select elementary.edr.dbt_artifacts
Here’s how that combination helps, based on the sources: 1. What causes the error The BigQuery error: > Transaction is aborted due to concurrent update against table project`.elementary.dbt_models` happens when multiple dbt runs finish at the same time and each run’s Elementary 
on-run-end
hook tries to update the same artifacts tables (
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc.) concurrently. [BQ concurrent runs] 2. Step 1 – Disable artifact autoupload in hooks You configure:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
This tells Elementary’s 
on-run-end
hook to skip the artifacts update step (the part that writes to 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc.). The hooks still run, but they no longer perform that heavy, conflict‑prone metadata update. [On‑run‑end hooks; BQ concurrency workaround] As a result, your normal dbt runs (including many in parallel) stop writing to those artifacts tables, which removes the main source of concurrent updates. 3. Step 2 – Centralize artifacts updates in one job After disabling autoupload, Elementary recommends that you update artifacts manually when needed, for example:
Copy code
dbt run --select elementary.edr.dbt_artifacts
run as a single job (e.g., a dedicated Airflow task or CI step) at a time you choose—typically on deploy or on a schedule. [On‑run‑end hooks; Reduce on‑run‑end time; BQ concurrency workaround] Because this job is not run in parallel with other artifact‑writing jobs, there is no competing transaction trying to update 
dbt_models
/ 
dbt_tests
at the same time—so the BigQuery “transaction aborted due to concurrent update” error no longer occurs for those tables. So, to answer your question precisely: • 
dbt run --select elementary.edr.dbt_artifacts
does not inherently make updates “safe”. • It prevents the concurrent transaction error when used as the only way artifacts are updated, after you’ve disabled automatic artifact uploads in the hooks. That design ensures there is just one controlled writer to the artifacts tables instead of many concurrent writers. [BQ concurrent runs; Airflow/concurrency guidance] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai what if I built a separate DAG that runs the dbt run --select elementary.edr.dbt_artifacts but it runs with another job ?
k
If that separate DAG can run at the same time as other dbt jobs that write to the same Elementary artifacts tables, you can still get the same concurrent transaction errors. From the sources, the pattern that avoids conflicts is: 1. Disable artifact autoupload in hooks, so regular dbt runs stop touching 
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc.:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
[BQ concurrent runs] 2. Have exactly one job that updates artifacts, e.g. a dedicated DAG/task that runs:
Copy code
dbt run --select elementary.edr.dbt_artifacts
and make sure this job is not competing with any other job that also updates those tables. [Disable autoupload workaround; Airflow concurrency guidance] So if your new DAG: • Runs 
dbt run --select elementary.edr.dbt_artifacts
, and • Other dbt runs (with hooks still updating artifacts) can run at the same time, then you still have multiple writers and can still hit the BigQuery “transaction is aborted due to concurrent update” error. To align with the documented workaround: • Set 
disable_dbt_artifacts_autoupload: true
so normal runs don’t update artifacts. • Use your separate DAG as the only place that runs 
dbt run --select elementary.edr.dbt_artifacts
, and schedule/serialize it so it doesn’t overlap with any other artifacts-writing process. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai does anything else write to the artifacts table ?
k
Mostly no—*only Elementary’s own artifacts models and hooks write to the artifacts tables*. From the docs: • The artifacts tables (
dbt_models
, 
dbt_tests
, 
dbt_sources
, 
dbt_exposures
, 
dbt_metrics
, 
dbt_snapshots
, 
dbt_seeds
, 
dbt_columns
, 
dbt_groups
) are dbt models in the Elementary package. They are populated either: ◦ by the 
on-run-end
hook when artifact autoupload is enabled, or ◦ when you explicitly run the models, e.g. 
dbt run --select elementary.edr.dbt_artifacts
. [dbt artifacts] • These metadata tables are “regular tables” that Elementary updates via 
truncate and insert
when those models run. [On‑run‑end hooks] The knowledge sources do not mention any other writers (e.g., dbt core itself or other packages) updating these Elementary artifacts tables. All described writes come from: 1. Elementary’s 
on-run-end
hooks (when not disabled), and 2. Manual runs of the Elementary artifacts models (
dbt run --select elementary
or 
--select elementary.edr.dbt_artifacts
). [On‑run‑end hooks; dbt artifacts] So, if you disable artifact autoupload and only run 
elementary.edr.dbt_artifacts
in a single controlled job, that job should be the only thing writing to those artifacts tables. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai difference between dbt run --select elementary.edr.dbt_artifacts vs dbt run --select elementary
k
Based on the knowledge sources, here's the difference: dbt run --select elementary.edr.dbt_artifacts This runs only the artifacts (metadata) sub-group of Elementary models — specifically the tables that describe your dbt project structure: • 
dbt_models
, 
dbt_tests
, 
dbt_sources
, 
dbt_exposures
, 
dbt_metrics
, 
dbt_snapshots
, 
dbt_seeds
, 
dbt_columns
, 
dbt_groups
This is the narrower, recommended command when you've disabled autoupload and just want to keep project metadata current. [dbt artifacts; Slack manual artifacts run] dbt run --select elementary This runs all Elementary models, which includes: • The artifacts/metadata models (same as above), plus • Other Elementary models (e.g. result processing models, metadata store, etc.) This is the broader command, typically used on first install or after a minor version upgrade of the Elementary package when schema changes may affect more than just the artifacts tables. [dbt artifacts; Slack when to run elementary] Which should you use? • If your goal is just to keep project metadata up to date and avoid concurrency issues → use 
elementary.edr.dbt_artifacts
(narrower, faster). • If you are installing Elementary for the first time or upgrading to a new minor version → use 
--select elementary
to ensure all model/schema changes are applied. [dbt artifacts] In your specific case (deploy-time refresh to avoid BigQuery concurrent update errors), 
dbt run --select elementary.edr.dbt_artifacts
is sufficient. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai currently elementary is only deployed in dev, but the artifacts upload is off what would be the best approach ?
k
Based on the conversation so far and the knowledge sources, here's what I can say: If Elementary is only deployed in dev and 
disable_dbt_artifacts_autoupload: true
is already set, the key things to be aware of are: 1. Artifacts must be run manually to populate metadata tables Since autoupload is off, the artifacts tables (
dbt_models
, 
dbt_tests
, 
dbt_sources
, etc.) will only be populated when you explicitly run:
Copy code
dbt run --select elementary.edr.dbt_artifacts
or
Copy code
dbt run --select elementary
If this hasn't been run in your dev environment, those tables will be empty or stale, and Elementary reports/alerts won't reflect your current project structure. [dbt artifacts; BQ concurrency workaround] 2. Recommended approach for dev-only setup • Run 
dbt run --select elementary
once on initial setup to initialize all Elementary models and tables. • After that, run 
dbt run --select elementary.edr.dbt_artifacts
whenever your project structure changes (new models, tests, sources, config changes). [dbt artifacts] • Since you're only in dev (single environment, likely no parallel runs), concurrency is less of a concern — but keeping autoupload disabled and running artifacts manually is still the cleaner pattern. 3. When you move to production When you eventually deploy to production (with Airflow + BigQuery), the pattern you've already discussed applies: • Keep 
disable_dbt_artifacts_autoupload: true
. • Run 
dbt run --select elementary.edr.dbt_artifacts
in a single, controlled job (e.g., on deploy or via a dedicated Airflow DAG/task), not in parallel with other dbt jobs. [BQ concurrent runs] The knowledge sources don't provide specific guidance for a dev-only scenario beyond this, so I'd recommend following the same pattern you've already established — it's consistent with Elementary's documented approach. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai what table do I check to see if artifacts have been uploaded ?
k
To verify that artifacts have been uploaded, you should query the metadata (artifacts) tables in your Elementary schema. The most relevant ones to check are: • 
dbt_models
– metadata about all models in your project • 
dbt_tests
– metadata about all tests • 
dbt_sources
– metadata about all sources • 
dbt_exposures
– metadata about exposures • 
dbt_snapshots
– metadata about snapshots • 
dbt_seeds
– metadata about seeds If artifacts have been uploaded successfully, these tables should contain rows reflecting your current dbt project structure. If they are empty or show a stale 
generated_at
timestamp, artifacts have not been uploaded (or not recently). [dbt artifacts; package models] A quick check would be:
Copy code
SELECT COUNT(*), MAX(generated_at) FROM <your_elementary_schema>.dbt_models;
SELECT COUNT(*), MAX(generated_at) FROM <your_elementary_schema>.dbt_tests;
• If 
COUNT(*)
is 0 → artifacts have not been uploaded. • If 
generated_at
is old → artifacts are stale and need to be refreshed with 
dbt run --select elementary.edr.dbt_artifacts
. The 
generated_at
column is present on all metadata tables and represents the last update time, so it's a reliable indicator of when artifacts were last uploaded. [dbt_models schema; dbt_seeds schema] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai [{ "f0_": "2136", "f1_": "2026-03-30 194224" }]
k
It looks like you're sharing the result of a query against one of the Elementary metadata tables — likely something like:
Copy code
SELECT COUNT(*), MAX(generated_at) FROM <your_elementary_schema>.dbt_models;
Based on the output: • 
f0_
= 2136 → There are 2,136 rows in the table, meaning artifacts have been uploaded and 
dbt_models
is populated. • 
f1_
= 2026-03-30 1942*24* → The last upload was on March 30, 2026 at 19:42 UTC. This confirms that your artifacts have been uploaded and the metadata tables are populated. [dbt_models schema] You can do the same check on 
dbt_tests
and 
dbt_sources
to confirm those are also up to date:
Copy code
SELECT COUNT(*), MAX(generated_at) FROM <your_elementary_schema>.dbt_tests;
SELECT COUNT(*), MAX(generated_at) FROM <your_elementary_schema>.dbt_sources;
👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai so do I still have to manually update the metadata ?
k
Based on your query results showing 2,136 rows with a 
generated_at
of 2026-03-30, your artifacts are currently populated. However, whether you need to manually update them going forward depends on your setup: Yes, you still need to manually update artifacts when your project structure changes, because you have 
disable_dbt_artifacts_autoupload: true
set (or plan to set it). This means the on-run-end hook will no longer automatically refresh those tables. From the docs, you should run:
Copy code
dbt run --select elementary.edr.dbt_artifacts
whenever: • You add or remove models, tests, or sources • You change configurations (severity, tags, owners, etc.) • You upgrade to a new minor version of the Elementary package [Control on‑run‑end time; dbt artifacts] You do not need to run it on every dbt run — only when your project structure actually changes. Since your current setup already triggers 
dbt run --select elementary
on Docker image rebuilds (which are tied to changes in 
dbt_project.yml
, 
packages.yml
, etc.), that should cover most cases where a metadata refresh is needed. [Slack manual artifacts run] The risk of not updating is that Elementary reports and alerts may show outdated or missing models/tests if your project changes but artifacts aren't refreshed. [dbt artifacts] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
what was the time it was last updated ? @kapa.ai
k
Based on the query result you shared earlier in our conversation, the artifacts were last updated at: 2026-03-30 1942*24 UTC* That was the 
MAX(generated_at)
value returned from your 
dbt_models
table. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai how frequently would the artifacts upload have to be run ?
k
Based on the knowledge sources, there is no fixed frequency requirement — artifacts only need to be updated when your project structure changes, not on a time-based schedule. Specifically, you should run:
Copy code
dbt run --select elementary.edr.dbt_artifacts
in these situations: • On initial install of the Elementary package • After minor package upgrades • When project structure/config changes, such as: ◦ Adding or removing models, tests, or sources ◦ Changing configurations (severity, tags, owners, etc.) [dbt artifacts; Control on‑run‑end time] For most teams, this means running it on deploy (e.g., when merging to main), which aligns with your current setup of triggering it on Docker image rebuilds. You do not need to run it on every dbt run or on a recurring hourly/daily schedule unless your project structure changes frequently. [Control on‑run‑end time] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
34 Views
Open in Slack
PreviousNext
Powered by