Hello Team, I am encountering persistent issues with on run end hooks with -

The query too large

in multiple env. This is BQ+DBT I have set these vars in dbt project yml config

# elementary limits
  dbt_artifacts_chunk_size: 500
  insert_rows_method: "chunk"
  query_max_size: 1000000
  disable_dbt_columns_autoupload: true

I did a full reset as well multiple times. I am on version -

elementary-data[bigquery]==0.16.1

Can you share the exact config? It needs to sit under vars like this:

vars:
  # elementary limits
  dbt_artifacts_chunk_size: 500
  insert_rows_method: "chunk"
  query_max_size: 1000000
  disable_dbt_columns_autoupload: true

I have like that as you mentioned. Reducing

dbt_artifacts_chunk_size

will help with -

The query is too large. The maximum standard SQL query length is 1024.00K characters, including comments and white space characters.

That's the idea yes. You can have a look at the actual query being executed in your target folder

It is publishing the job_id, but since its not able to load it, its not collecting the logs. My project is really big, it has north of 2k models + views. While reading the git history I think I am stuck in the bottleneck for these 2 variables -

insert_rows_method

and

query_max_size

I would say try around and play with the chunk_size, at some point it should work. Just make sure the performance is still ok. See more info here: https://github.com/elementary-data/elementary/issues/1689 (if you haven't) Not sure why you can't find the query in the logs. You should be able to find the logs in the gcp console for that error. Alternatively you should find the actual query being executed in your target folder. Hope it helps! Let me know 🙂

I think I found the issue and created a PR for the fix. Can you please review and approve? Trying to get some eyes from the maintainers so that this can be merged and released. Link - https://github.com/elementary-data/dbt-data-reliability/pull/763

Amazing! 🙂 We will have a look at it, but highly appreciate it

I am testing it now, do you think any var changes from here that needs to be changed?

# elementary limits
  dbt_artifacts_chunk_size: 150
  insert_rows_method: "chunk"
  query_max_size: 100000
  disable_dbt_columns_autoupload: true

No I think these should be fine as is!

Yes, I don't see the error further. Its working properly in prod. I move the package to git link with the latest # available. Thanks for the same.

Amazing! Thanks for letting me know 🙂