< kapa ai> I have dbt elementary package running in both `pr Elementary Community #ask-elementary-ai

<@U088WCP3B4P> I have dbt elementary package runni...

Parkash Pant

04/15/2026, 5:34 PM

@kapa.ai I have dbt elementary package running in both

pre

and

prod

environment. When I run

dbt build --select dbt_models

pre

, it creates new entry for each dbt model with different

generated_at

but in prod, it doesn't add any redundant record and only add records, if it found new model.

kapa.ai

04/15/2026, 5:35 PM

The behavior you're observing is by design and relates to how Elementary's

dbt_models

artifact table works. How dbt_models works The

dbt_models

table (along with

dbt_tests

dbt_sources

dbt_exposures

dbt_metrics

) is not an append-only table. It is designed to always reflect the latest state of your dbt project. On each run, Elementary performs a delete-and-insert (or merge) operation to replace existing rows with updated ones — it does not accumulate historical records. [dbt artifacts behavior] Why the behavior differs between pre and prod Starting from Elementary 0.15, the artifact models use hashes (dbt model hashes) to detect if a model has changed. If nothing has changed in your project, Elementary will skip updating some or all dbt artifact rows. [Elementary 0.15 behavior] So the difference you're seeing likely comes down to: • In prod
: Elementary detects no changes (same model hashes), so it skips inserting redundant records — only new models get added. • In pre
: Something is causing the hashes to differ on each run (e.g., different environment config, target name, schema, or metadata), so Elementary treats models as "changed" and re-inserts them with a new

generated_at

. What to check 1. Are your environments writing to the same Elementary schema? If multiple environments write to the same schema, they can overwrite each other's data. Each environment should have its own dedicated Elementary schema. [multi-project issue] 2. Check your
dbt_project.yml
flags — if you have conditional logic like

disable_dbt_artifacts_autoupload

tied to environment names, make sure it's consistent. [artifact upload config] 3. Consider disabling Elementary in non-prod environments if you don't need it there. A common pattern is:

Copy code

models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.name == 'prod' }}"

[disable in non-prod] Or, if you want to keep Elementary enabled but skip the hooks in `pre`:

Copy code

vars:
  disable_run_results: "{{ target.name != 'prod' }}"
  disable_tests_results: "{{ target.name != 'prod' }}"
  disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"
  disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"

The knowledge sources don't provide the exact internal logic that determines when a hash is considered "changed" in

pre

prod

, so I'd recommend checking your environment-specific configurations for differences that could affect model hashes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Parkash Pant

04/15/2026, 5:41 PM

@kapa.ai In my pre, I see metadata hash field exactly same for the redundant records.

kapa.ai

04/15/2026, 5:42 PM

That's an important detail. If the

metadata_hash

is identical across the duplicate rows in

pre

, this points to a different root cause than hash-based change detection. Based on the knowledge sources, this is a known race condition bug in Elementary's

delete_and_insert

macro. When multiple dbt runs (or jobs) trigger the Elementary

on-run-end

hook concurrently, the same metadata can be inserted multiple times into

dbt_models

, resulting in duplicate rows with the same

unique_id

and

metadata_hash

but different

generated_at

values. [duplicate entries issue] This is especially common when: • Multiple dbt commands run in parallel (e.g., on Airflow or another orchestrator) •

edr monitor

and

dbt run

are triggered at the same time [duplicate entries issue] Workarounds 1. Serialize your Elementary-related steps — avoid running multiple jobs that trigger Elementary artifact updates concurrently. [prevent duplicates] 2. Disable automatic artifact uploads and manage them centrally usingvars disable_dbt_artifacts_autoupload: trueThen run artifacts refresh explicitly once per deploy:dbt run --select elementary.edr.dbt_artifacts[prevent duplicates] 3. Clean up existing duplicates by keeping only the latest row per `unique_id`:create or replace table elementary.dbt_models as ( select * from elementary.dbt_models qualify row_number() over (partition by unique_id order by generated_at) = 1 );The same pattern applies to

dbt_tests

dbt_sources

dbt_exposures

, and

dbt_columns

. [duplicate cleanup] 4. Set
cache_artifacts = False
to force a full replace on every run (note: this increases

on-run-end

duration)vars cache_artifacts: false[cache_artifacts workaround] The Elementary team has acknowledged this as an open bug tied to a race condition in the

delete_and_insert

macro, with no documented final fix yet. [open issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Parkash Pant

04/15/2026, 5:44 PM

@kapa.ai I am just running dbt build --select dbt_models and no one else is running elementary package.

kapa.ai

04/15/2026, 5:45 PM

That's a helpful clarification. If you're running a single

dbt build --select dbt_models

with no concurrent runs, the race condition explanation doesn't apply here. Based on the knowledge sources, there is one other documented cause of duplicates in

dbt_artifacts_hashes

and related tables that occurs even with a single thread/run: overriding Elementary's model materialization in your project-level config. One community member discovered that adding

+materialized: incremental

at the project level in

dbt_project.yml

was accidentally overriding Elementary's

dbt_artifacts_hashes

model (which is designed to be a view), turning it into an incremental table. This caused duplicates to accumulate in the underlying incremental tables. [dbt_artifacts_hashes duplicates] What to check: Look at your

dbt_project.yml

for any broad materialization overrides like:

Copy code

models:
  +materialized: incremental

If such a config exists at a level that affects the

elementary

package models, it could be causing this issue. The fix was to remove the override so that Elementary's models use their intended materializations. Quick fix to try: Run:

Copy code

dbt run -s elementary

This has been reported to clean up duplicates in the metadata tables. [duplicate cleanup] Beyond this, the knowledge sources don't document other causes of duplicates from a single, non-concurrent run. If the above doesn't apply to your setup, I'd recommend reaching out to the Elementary team directly with your

dbt.log

for further investigation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Parkash Pant

04/15/2026, 5:45 PM

@kapa.ai Our global materialization is table.

kapa.ai

04/15/2026, 5:46 PM

That's very likely the root cause of your issue. If you have a global materialization set like this in your `dbt_project.yml`:

Copy code

models:
  +materialized: table

This overrides Elementary's own model configs. Specifically, Elementary's

dbt_artifacts_hashes

model is designed to be a view, but a global

+materialized: table

(or

incremental

) will turn it into a table, causing duplicates to accumulate. [dbt_artifacts_hashes duplicates] Similarly, Elementary's incremental models (like

dbt_models

dbt_invocations

, etc.) can be broken when a global materialization override forces them to be rebuilt as plain tables instead of using their intended incremental logic. [dbt_invocations override] The Fix Scope your global materialization to your own project only, not the

elementary

package:

Copy code

models:
  your_project_name:       # <-- use your actual project/models directory name
    +materialized: table
  elementary:
    +schema: "elementary"  # elementary keeps its own materializations

This ensures your models are materialized as tables, while Elementary's models continue to use their intended materializations (incremental, view, etc.). [recommended config] After fixing the config, run a full refresh to clean up the existing duplicates:

Copy code

dbt run --select elementary --vars '{"elementary_full_refresh": "true"}'

[full refresh fix] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Parkash Pant

04/15/2026, 5:47 PM

@kapa.ai I meant for our own project, not elementary package. For our own project, we have set global materialization to table

kapa.ai

04/15/2026, 5:47 PM

Thanks for the clarification. If your global

+materialized: table

is already scoped to your own project (not affecting the

elementary

package), then that specific issue doesn't apply. However, based on the knowledge sources, there is still one important thing to verify: how exactly your
dbt_project.yml
is structured. Even when you intend to scope the config to your own project, if the

materialized: table

is placed at the top-level

models:

block rather than under your project's subdirectory key, it can still bleed into Elementary's models. For example, this is incorrect (affects all models including Elementary):

Copy code

models:
  materialized: table        # top-level, affects everything
  elementary:
    +schema: "elementary"

This is correct (scoped only to your project):

Copy code

models:
  your_project_name:         # your actual project/models directory name
    +materialized: table
  elementary:
    +schema: "elementary"

[dbt_invocations override] If your config is already correctly scoped and Elementary's models are unaffected, then the knowledge sources don't document another cause of duplicates from a single, non-concurrent

dbt build --select dbt_models

run. In that case, I'd recommend reaching out to the Elementary team directly with your

dbt_project.yml

and

dbt.log

for further investigation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Parkash Pant

04/15/2026, 5:48 PM

@kapa.ai We are using this structure. models: your_project_name: # your actual project/models directory name +materialized: table elementary: +schema: "elementary"

kapa.ai

04/15/2026, 5:48 PM

Thank you for confirming your config is correctly scoped. In that case, the global materialization override is not the cause of your duplicates. To be transparent: the knowledge sources I have available do not document a specific cause for duplicate records in

dbt_models

when: • Only a single

dbt build --select dbt_models

is run (no concurrency) • The

elementary

package materialization is not overridden • The

metadata_hash

is identical across duplicate rows The documented causes in the knowledge sources are: 1. Concurrent/parallel dbt runs triggering the

on-run-end

hook simultaneously [duplicate entries issue] 2. Materialization overrides affecting Elementary's models [dbt_artifacts_hashes duplicates] Since neither applies to your situation, I'd recommend: 1. Run
dbt run -s elementary
as an immediate cleanup step — this has resolved duplicates for several community members [duplicate cleanup] 2. Contact the Elementary team directly with your

dbt.log

, your

dbt_project.yml

, and a sample of the duplicate rows (including

unique_id

metadata_hash

, and

generated_at

) for a proper investigation, as this appears to be an edge case not fully covered in the available documentation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.

29 Views

Open in Slack

Previous Next