Would independently using

dbt_artifacts

package negative impact performance or cause crash of elementary package?

Hey @Jintae Park, Not suppose to. Have you experienced otherwise ?

@Daniel Pollak Thanks for replying back. I have been trying to diagnose the long runtime & on-end-hook erroring out and it seems like high number columns (~85K) on my project is the cause. Anything you can recommend me for this case? We are currently running on BigQuery + dbt 1.6.2

dbt run -s elementary

takes about 30+ minutes due to

dbt_columns

materializing

00:33:59  2 of 30 OK created sql incremental model jintae_park_elementary.dbt_columns .... [CREATE TABLE (0.0 rows, 0 processed) in 1949.11s]

And running

dbt test

seems to errors out at on-run-end steps - I actually think it is query length because error went away after commenting most of the values.

01:15:29    on-run-end failed, error:
 Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex.

/* {"app": "dbt", "dbt_version": "1.6.2", "profile_name": "data_office", "target_name": "local-dev", "connection_name": "master"} */

    
       insert into `branch-dbt-dev`.`jintae_park_elementary`.`dbt_columns__tmp_20240403011402048064`
         (metadata_hash) values
    ('00016f8d146ede445cd52c9b0b4d5385'),('0002b6d1c0d8c5bc5fdf9a0fde681286'),('000347e0ba8ea5225cd6d4eb7abcd944'),('0005b677282d1e1a354d5677f9f5792d'),('000648efd04f33d8791a98a8f983825f'),('00068ae2d772839ae42bddfec1eeb3f9'),('0006d483578ddcfa11c46aab664a634b'),('00070321119627ad4ade0cd1fa7c2feb'),('0007e9da7fd6ea872e196be3e5f16298'),('0009e0e4838676e799fc4ca97dbbd6c4'),('000bf24a81a2ad00dc8cbcb720ee9ec2'),('000e4cef39d3369e0e463289a7a0cfd8'),('0010d249d7c256f4fb51616fcd2a9aa3'),('0010de4825fdb29054e6cce8f22c5f71'),('0010e0ce775739c03c920efb459864e3'),('0011689816704842ae08e3ae81f2e6cc'),('0011ad9f733dc4c387a047ce78dff5e4'),('001304f90e5601cb4ad292287137d1db'),('00134821e47c2ecef317092b40f0d616'),('0013648fc56e0128be3f1e7b065a240c'),('00138a3ea4d8ab4eaf39e01897a4c42a'),('0013a039bf60ae11ec471aa5b2873256'),('0015976ad0fcfa6a93b981865a8dbb07'),('00161a976dd4c9fa4770c6e7b1187841'),('00178744adc6165b5e3db1b06dcf0777'),('0017f53db834317e42597c87c324ea1b'),('0019506841c7a667dc2016f37a341455'),('0019bdc386e50ea0feddff4869dac5de'),('001ab4de713cb015b8009d8f1f39cb44'),('001c5b880e40947efb4327c45ff006ea'),('001db6d6759c6f0b8606a2e146b29178'),('001dfd7addc825030b187eb891efad9e'),('001f07c97e2449a51ec5aae67ca8473a'),('001fd1aaed244e50516737a34f34c836'),('00220b9b5a0ad5ce04200d18522bd147'),('0023b4e1867cbef182dddaf054ebd20e'),('0024533b9b24487cb8dd9861c3a5ffc7'),('0026723d58669dc0975459f8af90e5a2'),('0027181a34efd5ad087c758d3ee9724c'),('002a2deb3499c3e3030679880fc0dd02'),('002b75f3e9305ba129aa3b32c8e27d10'),('002da332523a5b159e6219742ae24cd0'),('002de78749c046e136611b6fa192fa47'),('002e2710e77e7a526098c3573e4d6fbf'),('0030f1570712fd9b2717a8189ee53b21'),('0031b000174afbda2a3097d8107c4862'),('00324855a6041665864822ea1d89ac74'),('0035796d157f3852bcdf744aaff5cf07'),('00357e1b87d6ded41ae6725e0254b0c0'),('00365cf913d4b12e167e74d5e3cbc40c'),('00375c3bece0008aaeaa22a803f57628'),('0037b20e781baec57f23b632323258a7'),('003890283ee45630f106f668fe56c5a0'),('0039876db2d25f0f0b6fec37b4c725c8'),('0039dddaf38ec1cf251df5cd43e21966')

...

Curious if this managed to fix the problem? Please!

@Josh Ended up disabling

dbt_columns

upload entirely by setting

disable_dbt_columns_autoupload: true

We have a daily workflow that runs this on the side.

insert_rows_method: max_query_size
  query_max_size: 10000
  disable_dbt_columns_autoupload: true

I have similar issue - running dbt test locally.

dbt test --profiles-dir . --vars '{"environment":"xyz-dev"}' --select models/sdm/general/_general_sdm__models.yml

bigquery: 1.7.6, dbt core 1.7.9

09:46:00  Database error while running on-run-end

09:46:00  Completed with 1 error and 0 warnings:

09:46:00    on-run-end failed, error:

Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex.

I inserted the below settings into config.yml

insert_rows_method: max_query_size
  query_max_size: 10000
  disable_dbt_columns_autoupload: true

did not helped. Question: It there a possibility to set this "error" to a warning? (We are using Airflow for the orchestration. The error will stop the DAG.)

Thanks for this it seems to work but the end hook is just taking ages now 😬