<@U088WCP3B4P> My dbt models table has multiple du...
# ask-elementary-ai
l
@kapa.ai My dbt models table has multiple duplicated rows with the same unique_id 140010 Database Error in model dbt_models (src/models/monitoring/artifacts/dbt_models.sql) [DELTA_MULTIPLE_SOURCE_ROW_MATCHING_TARGET_ROW_IN_MERGE] Cannot perform Merge as multiple source rows matched and attempted to modify the same target row in the Delta table in possibly conflicting ways. By SQL semantics of Merge, when multiple source rows match on the same target row, the result may be ambiguous as it is unclear which source row should be used to update or delete the matching target row. You can preprocess the source table to eliminate the possibility of multiple matches. Please refer to https://docs.databricks.com/delta/merge.html#merge-error
k
A message from kapa.ai
This error is caused by duplicate rows in Elementary’s
dbt_models
artifacts table (same
unique_id
with different
generated_at
), which then cause a Databricks Delta MERGE to see “multiple source rows matching target row.” From the shared issues and Slack threads, there are two main causes: 1. Concurrent artifact uploads / concurrent dbt runs - When multiple dbt runs (or
edr monitor
/
edr report
) run at the same time, each may try to update the Elementary artifact models, leading to duplicate records per
unique_id
in
dbt_models
,
dbt_tests
,
dbt_sources
, etc. [[dup bug issue](https://github.com/elementary-data/elementary/issues/1420); [concurrency thread](https://elementary-community.slack.com/archives/C02CTC89LAX/p1672942935757879?thread_ts=1672942935.757879&amp;cid=C02CTC89LAX); [Databricks conflicts](https://elementary-community.slack.com/archives/C02CTC89LAX/p1694431548484389?thread_ts=1694431548.484389&amp;cid=C02CTC89LAX)] - This is especially common on Databricks / Delta where concurrent DELETE/TRUNCATE + INSERT patterns hit
DELTA_CONCURRENT_DELETE_READ
or “multiple source rows” MERGE errors. [[Databricks error](https://elementary-community.slack.com/archives/C02CTC89LAX/p1747669372225679?thread_ts=1747669372.225679&amp;cid=C02CTC89LAX)] 2. The way artifacts tables are maintained (cache / delete_and_insert race) - There’s an open issue that duplicates appear in
dbt_models
and other artifacts tables when models are run in parallel, likely due to a race in the
delete_and_insert
macro / cache behavior. [[dup bug issue](https://github.com/elementary-data/elementary/issues/1420)] ### How to fix your current
dbt_models
table As a cleanup step, users are successfully de‑duplicating the artifacts by keeping only the latest row per `unique_id`:
Copy code
sql create or replace table elementary.dbt_models as (     select *     from elementary.dbt_models     qualify row_number() over (partition by unique_id order by generated_at) = 1 );
This pattern is also used for other artifact tables (
dbt_tests
,
dbt_sources
,
dbt_exposures
,
dbt_columns
) when they suffer from the same issue. [[dup cleanup workaround](https://github.com/elementary-data/elementary/issues/1420)] After running this, rerun your
dbt_models
model / pipeline and the MERGE error should stop (until duplicates accumulate again). ### How to reduce / prevent the duplicates Based on the threads: 1. Avoid concurrent artifact updates - Don’t run multiple jobs that trigger Elementary artifact updates at the same time (e.g. multiple
dbt run -s elementary
, or
edr monitor
and
edr monitor send-report
running concurrently from the same project). [[concurrency explanation](https://elementary-community.slack.com/archives/C02CTC89LAX/p1672942935757879?thread_ts=1672942935.757879&amp;cid=C02CTC89LAX)] - If you’re on Airflow or another orchestrator, serialize the Elementary‑related steps. 2. Disable automatic artifact uploads and manage them centrally - One suggested mitigation (especially in concurrent environments) is to turn off automatic artifact uploads from on‑run‑end hooks using
disable_dbt_artifacts_autoupload = true
, and instead run:
Copy code
bash      dbt run --select elementary.edr.dbt_artifacts
once per deploy to refresh artifacts in a controlled, non‑concurrent job. [[freshness dup discussion](https://elementary-community.slack.com/archives/C02CTC89LAX/p1754991499736459?thread_ts=1754991499.736459&amp;cid=C02CTC89LAX)] 3. Ensure Elementary + dbt versions are aligned / up to date - Some threads mention changes to
dbt_models
in newer Elementary dbt packages and improvements around concurrency; upgrading both the Elementary dbt package and the
edr
CLI to recent aligned versions is recommended, although the duplication race is still tracked as an open issue. [[serverless Databricks thread](https://elementary-community.slack.com/archives/C02CTC89LAX/p1747669372225679?thread_ts=1747669372.225679&amp;cid=C02CTC89LAX); [dup bug issue](https://github.com/elementary-data/elementary/issues/1420)] If, after de‑duping and serializing artifact updates, you still see new duplicates and MERGE errors, the remaining behavior is likely tied to the open race condition mentioned in the GitHub issue; there’s no documented final fix yet in the sources provided.