Hi, First, thanks for being involved in developin...
# community-support
c
Hi, First, thanks for being involved in developing this package. I think this topic has occurred before and if someone could lead me to the outcome that would be awesome thanks. So, we have duplicates in our dbt_artifacts_hashes table. This appears to have no function, the duplicates, except we have proven that is slows down execution.
Copy code
select artifacts_model, metadata_hash, count(*) total
from `uat_elementary`.`dbt_artifacts_hashes`
group by artifacts_model, metadata_hash
order by total desc
Do you know how the metadata_hash generates the hash please? It appears to use the metadata.json so does that mean we need to add more metadata in our ymls to make them unique? Do you know how the dbt_artifacts_hashes table gets populated please? I can see the table get read but not written to: https://github.com/elementary-data/dbt-data-reliability/blob/26e24f94cd7dd27cd237d84be268bc51646e1e32/macros/edr/system/hooks/on_run_end.sql#L9 https://github.com/elementary-data/dbt-data-reliability/blob/master/macros/edr/dbt_artifacts/upload_dbt_artifacts.sql#L18 This does not appear to be a thread issue as only 1 thread produces the duplicates. Our versions: dbt-core >= 1.7.0, < 1.8.0 dbt-databricks >= 1.7.0, < 1.8.0 packages: - package: dbt-labs/dbt_utils version: 1.2.0 - package: calogica/dbt_expectations version: 0.10.3 - package: elementary-data/elementary version: 0.15.2 Thanks for any help
💡 1
I have discovered how the dbt_artifacts_hashes view is created and we have probably overridden it with a table by adding "+materialized: incremental" in our project settings. https://github.com/elementary-data/dbt-data-reliability/blob/master/models/edr/dbt_artifacts/dbt_artifacts_hashes.sql . Not sure how much that will improve the performance as there are still duplicates created in the incremental tables that create the view, for example: https://github.com/elementary-data/dbt-data-reliability/blob/master/models/edr/dbt_artifacts/dbt_models.sql
We removed "+materialized: incremental" from our project level config and it seemed to fix the performance issue by taking dbt_artifacts_hashes back to a view.