why do we have double entry in the metadata collec...
# ask-elementary-ai
j
why do we have double entry in the metadata collected by Elementary? e.g.
dbt_tests
k
A message from kapa.ai
Duplicate entries in metadata tables like
dbt_tests
in Elementary are a known issue, most commonly caused by running dbt tests or models in parallel. When multiple dbt commands (such as
dbt test
or
dbt run
) are executed simultaneously, the Elementary
on-run-end
hook—which is responsible for updating metadata tables—can be triggered concurrently. This can result in race conditions where the same metadata is inserted multiple times, leading to duplicate rows in tables such as
dbt_tests
,
dbt_sources
, and others. This issue has been reported by several users and is documented in both GitHub issues and community discussions. For example, one GitHub issue describes how running tests/models in parallel as separate queries leads to duplicate entries in artifact tables, which then causes duplicate records in reports and alerts. The Elementary team has acknowledged this as a race condition in the
delete_and_insert
macro and is working on a fix. As a workaround, users have been advised to periodically deduplicate these tables or to avoid running dbt jobs in parallel if possible. Another workaround is to run
dbt run -s elementary
manually, which can clean up the duplicates in the metadata tables, though this is not always a permanent solution if parallel runs continue to occur. There is also a suggestion to set the variable
cache_artifacts = False
to force a full replace of artifacts on every run, but this can increase the duration of the
on-run-end
hook and may not be suitable for all environments [Duplicate entries in the report (GitHub)](https://github.com/elementary-data/elementary/issues/1420), [Slack thread on duplicates](https://slack.com/archives/C02CTC89LAX/p1683632760607909), [Another Slack thread on duplicates](https://slack.com/archives/C02CTC89LAX/p1695056085072099). In summary, double entries in
dbt_tests
and similar metadata tables are typically due to concurrent dbt runs causing the Elementary hooks to insert duplicate metadata. The Elementary team is aware of the issue and is working on a more robust fix. For now, avoid parallel runs or periodically deduplicate the tables as a workaround.