Hi, running into Bigquery quota issues with elementary data. we are working with over 2k dbt assets and process petabytes to give a sense of scale 1. what is the defaults for these settings? insert_rows_method: dbt_artifacts_chunk_size: query_max_size: 2. what should we use to determine appropriate values for these parameters? thank you!!

Hey Chris! Try these configurations in your

dbt_project.yml

, as suggested here:

vars:
  insert_rows_method: "chunk"
  dbt_artifacts_chunk_size: 500
  query_max_size: 100000

Making sure, were you able to run Elementary before?

sorry for the delayed response. for us the issue isnt about "too many queries or query is too complex". we are simply sending TOO much insert statements

Absolutely 🙂 • dbt_artifacts_chunk_size: 5000 rows per insert • query_max_size: 250000 characters for BigQuery (1000000 for most other databases) The

insert_rows_method

var has been removed starting dbt package 0.16.2. Let me know if you find a way to make it work with that amount of data. If not, we can try thinking of another solution for this.