Noya Offer
05/12/2025, 9:16 AMkapa.ai
05/12/2025, 9:16 AMkapa.ai
05/12/2025, 9:16 AMyaml vars: insert_rows_method: "chunk" dbt_artifacts_chunk_size: 1000 # You can adjust this number based on your needs query_max_size: 1000000 # For BigQuery, this should be below the documented query size limit This configuration tells Elementary to batch insert rows in chunks (e.g., 1000 at a time), which is much faster than inserting one row per query. Users have reported that this resolves the issue of slow inserts and avoids hitting BigQuery's query size limits. If you still experience issues, try lowering the dbt_artifacts_chunk_size further, as sometimes very large rows can still cause queries to exceed limits even with chunking enabled. For advanced use cases or very large data volumes, Elementary also supports inserting via a GCP Cloud Function and Pub/Sub, which can further improve performance, but chunked inserts are usually sufficient for most users [Add support for inserts via GCP cloud function and pub/sub](https://github.com/elementary-data/dbt-data-reliability/pull/670). References: - [Github Issue: "Too many queries or query is too complex" issue](https://github.com/elementary-data/elementary/issues/1689) - [Slack: Query size and chunking for BigQuery](https://slack.com/archives/C02CTC89LAX/p1696005981230079) If you continue to see one-row-at-a-time inserts after setting these variables, ensure you are using a recent version of the Elementary dbt package, as earlier versions had bugs where chunking was not always respected [Slack: Chunking bug fix in 0.14.1](https://slack.com/archives/C02CTC89LAX/p1706626800346159).