@kapa.ai We are using elementary for monitoring and have multiple processes which write to the monitoring tables. We are seeing duplicate entries in those tables and want to avoid those. Additional Context: - We have multiple dbt runs/processes running concurrently or in quick succession - The duplicates appear in all of the tables like

dbt_sources, dbt_tests, dbt_models, dbt_snapshots, dbt_columns, dbt_seeds

- Our current elementary models are using the default table materialization strategy I was checking if there is any inbuilt deduplication mechanism built within elementary to deal with this, however I couldn't find any solution for this. I wanted to check if you have seen issues like this before and what is usually recommended in such scenarios. Some approaches I'm considering: 1. Create a dedup macro which can act as a post hook after every process runs to delete duplicates 2. Override elementary model configs to use incremental materialization with merge strategy Queries: - Is this a known issue with concurrent dbt runs? - Does elementary have recommended patterns for handling multiple parallel processes? - Are there specific unique_key combinations that work best for elementary's core tables? - What would be a better solution in this case to approach this issue? Would appreciate any guidance or best practices you can share!

A message from kapa.ai