Please answer <@U08S0DLE0KT>'s question - I've ad...
# ask-elementary-ai
n
Please answer @Anna Nguyen's question - I've added some elementary tests to some larger tables, in my dbt repository, but the insertion queries into test_result_rows is taking a really really long time. It's also triggering individual queries in gcp to insert into the table (seemingly) one row at a time? This happens after a dbt build of all the tables.
k
A message from kapa.ai
It sounds like you're experiencing slow performance when Elementary inserts test results into the test_result_rows table, and you notice that in GCP (likely BigQuery), the inserts are happening one row at a time, which is highly inefficient. This issue has been discussed in the Elementary community, especially for BigQuery users. By default, Elementary tries to insert many rows at once, but if not configured properly, or if certain limits are hit (like query size or row count), it may fall back to less efficient methods, such as inserting rows individually. Recommended Solution: You should configure Elementary to use chunked inserts, which batches rows into groups and inserts them together, significantly improving performance. You can do this by adding the following to your `dbt_project.yml`:
Copy code
yaml vars:   insert_rows_method: "chunk"   dbt_artifacts_chunk_size: 1000  # You can adjust this number based on your needs   query_max_size: 1000000         # For BigQuery, this should be below the documented query size limit
This configuration tells Elementary to batch insert rows in chunks (e.g., 1000 at a time), which is much faster than inserting one row per query. Users have reported that this resolves the issue of slow inserts and avoids hitting BigQuery's query size limits. If you still experience issues, try lowering the
dbt_artifacts_chunk_size
further, as sometimes very large rows can still cause queries to exceed limits even with chunking enabled. For advanced use cases or very large data volumes, Elementary also supports inserting via a GCP Cloud Function and Pub/Sub, which can further improve performance, but chunked inserts are usually sufficient for most users [Add support for inserts via GCP cloud function and pub/sub](https://github.com/elementary-data/dbt-data-reliability/pull/670). References: - [Github Issue: "Too many queries or query is too complex" issue](https://github.com/elementary-data/elementary/issues/1689) - [Slack: Query size and chunking for BigQuery](https://slack.com/archives/C02CTC89LAX/p1696005981230079) If you continue to see one-row-at-a-time inserts after setting these variables, ensure you are using a recent version of the Elementary dbt package, as earlier versions had bugs where chunking was not always respected [Slack: Chunking bug fix in 0.14.1](https://slack.com/archives/C02CTC89LAX/p1706626800346159).