Hello Elementary team - want to reach out with a s...
# community-supportj
Jintae Park
03/20/2024, 5:49 PMHello Elementary team - want to reach out with a setup question:
I've been going through the initial setup for Elementary OSS on our dbt core + BigQuery stack and was stumped on long runtime of on-run-end jobs.
Is it typical to have 30+minute runtime for project of our size? (roughly 1200+ models / 300+ tests)
Setup:
• dbt version 1.6.6
• bigquery adapter 1.6.7
• google-api-core 2.15.0 (using an older version due to some bug)
Details:
• For the initial
dbt run --select elementarydbt_columnsResources exceeded during query execution1️⃣ 1
Jintae Park
03/20/2024, 5:53 PMdbt run --select elementary Copy code
15:52:43 Running with dbt=1.6.6
15:52:43 Registered adapter: bigquery=1.6.7
15:52:43 Unable to do partial parsing because saved manifest not found. Starting full parse.
15:53:23 Found 1278 models, 31 snapshots, 307 analyses, 81 seeds, 5 operations, 355 tests, 421 sources, 0 exposures, 0 metrics, 1251 macros, 0 groups, 0
semantic models
15:53:23
15:53:27
15:53:27 Running 3 on-run-start hooks
15:53:27 1 of 3 START hook: branch_dbt.on-run-start.0 ................................... [RUN]
15:53:27 1 of 3 OK hook: branch_dbt.on-run-start.0 ...................................... [OK in 0.00s]
15:53:27 2 of 3 START hook: branch_dbt.on-run-start.1 ................................... [RUN]
15:53:31 2 of 3 OK hook: branch_dbt.on-run-start.1 ...................................... [SCRIPT (0 processed) in 4.02s]
15:53:31 3 of 3 START hook: elementary.on-run-start.0 ................................... [RUN]
15:53:31 3 of 3 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
15:53:31
15:53:31 Concurrency: 8 threads (target='local-dev')
15:53:31
15:53:31 1 of 30 START sql incremental model jintae_park_elementary.data_monitoring_metrics [RUN]
15:53:31 2 of 30 START sql incremental model jintae_park_elementary.dbt_columns ......... [RUN]
...
16:25:54 2 of 30 OK created sql incremental model jintae_park_elementary.dbt_columns .... [CREATE TABLE (0.0 rows, 0 processed) in 1943.16s]
...
16:25:56
16:25:56 Running 2 on-run-end hooks
16:25:56 1 of 2 START hook: branch_dbt.on-run-end.0 ..................................... [RUN]
16:25:56 1 of 2 OK hook: branch_dbt.on-run-end.0 ........................................ [OK in 0.00s]
16:26:10 2 of 2 START hook: elementary.on-run-end.0 ..................................... [RUN]
16:26:10 2 of 2 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
16:26:10
16:26:10
16:26:10 Finished running 15 incremental models, 1 table model, 14 view models, 5 hooks in 0 hours 32 minutes and 46.99 seconds (1966.99s).
16:26:11
16:26:11 Completed successfully
16:26:11
16:26:11 Done. PASS=30 WARN=0 ERROR=0 SKIP=0 TOTAL=30Jintae Park
03/20/2024, 5:56 PMOutputs from running short list of tests
Copy code
16:29:49 Running with dbt=1.6.6
16:29:49 Registered adapter: bigquery=1.6.7
16:29:56 Found 1278 models, 31 snapshots, 307 analyses, 81 seeds, 5 operations, 355 tests, 421 sources, 0 exposures, 0 metrics, 1251 macros, 0 groups, 0
semantic models
16:29:56
16:30:00
16:30:00 Running 3 on-run-start hooks
16:30:00 1 of 3 START hook: branch_dbt.on-run-start.0 ................................... [RUN]
16:30:00 1 of 3 OK hook: branch_dbt.on-run-start.0 ...................................... [OK in 0.00s]
16:30:00 2 of 3 START hook: branch_dbt.on-run-start.1 ................................... [RUN]
16:30:04 2 of 3 OK hook: branch_dbt.on-run-start.1 ...................................... [SCRIPT (0 processed) in 4.11s]
16:30:05 3 of 3 START hook: elementary.on-run-start.0 ................................... [RUN]
16:30:05 3 of 3 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
16:30:05
16:30:05 Concurrency: 8 threads (target='local-dev')
16:30:05
16:30:05 1 of 54 START test not_null_dim_address_pk_address ............................. [RUN]
...
16:30:14 53 of 54 PASS unique_fct_stripe_refund_pk_stripe_refund ........................ [PASS in 1.21s]
16:30:14
16:30:14 Running 2 on-run-end hooks
16:30:14 1 of 2 START hook: branch_dbt.on-run-end.0 ..................................... [RUN]
16:30:14 1 of 2 OK hook: branch_dbt.on-run-end.0 ........................................ [OK in 0.00s]
...
17:27:03 Database error while running on-run-end
17:27:03
17:27:03 Finished running 54 tests, 4 hooks in 0 hours 57 minutes and 7.16 seconds (3427.16s).
17:27:04
17:27:04 Completed with 1 error and 0 warnings:
17:27:04
17:27:04 on-run-end failed, error:
Resources exceeded during query execution: Not enough resources for query planning - too many subqueries or query is too complex.
17:27:04
17:27:04 Done. PASS=54 WARN=0 ERROR=1 SKIP=0 TOTAL=5529 Views