Noam Moskowitz
10/08/2024, 8:32 AMdbt-core == 1.7.3
dbt-bigquery == 1.7.3
elementary-data == 0.16.1
and my dbt package is with:
- package: elementary-data/elementary
version: 0.16.1
previously we used :
dbt-core == 1.6.1
dbt-bigquery == 1.6.1
elementary-data == 0.14.1
and my dbt package is with:
- package: elementary-data/elementary
version: 0.14.1
In my dbt_project.yml these are my vars:
query_max_size: 100000
insert_rows_method: "chunk"
dbt_artifacts_chunk_size: 500
The issue is the on-run-end hook which used to take 20 minutes and now it gets to 120.
1. This happens only when running the all_columns_anomalies test.
2. Dosent happen when running dbt run command.
3. There are multiple dbt.log files created during the on-run-end hook.
4. It looks like it inserts the whole history over and over again.
anyone has an idea how to decrease it without downgrading the version???
I need some help on this ASAP.