https://linen.dev logo
Join Slack
Powered by
After upgrading to `0.16.2`, `elementary.on-run-en...
# community-support
g
After upgrading to 
0.16.2
, 
elementary.on-run-end
hook seems to be taking ages (from a few seconds this now requires ~10-15 minutes!). Any clue how to solve this issue?
m
Hi @George Myrianthous Is it completed after 10-15 minutes? Are there any errors? Could you please share the logs for us to review?
g
yes, it is completed with no errors
the problem is the time required that seems to be unreasonable
m
I completely understand! We would like to look into the issue. Could you please share the logs from the dbt run? Just to clarify, is 
0.16.2
the version of the dbt package or the Python package? The latest version of the dbt package is 
0.16.4
, and the latest version of the Python package is 
0.16.2
.
g
We are using 
0.16.2
for both python and dbt packages
m
Did it start happening right after the version upgrade? What were the dbt and Python package versions prior to the upgrade?
g
0.14.1 dbt package, and 0.16.1 Python package
m
Got it 👌 The dbt logs are a great starting point for investigating this. Additionally, could you give upgrading to version 0.16.4 a try? Let me know if you still encounter the same issue 🙏
g
We have tried 0.16.4 already, and seems to be equally slow.
Here’s a snapshot of the logs:
Copy code
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] IMPORTANT - Starting from dbt 1.8, users must explicitly allow packages to override materializations.
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] Elementary requires this ability to support collection of samples and failed row count for dbt tests.
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] Please add the following flag to dbt_project.yml to allow it:
2025-02-18T08:46:42.921980853Z 

[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] flags:
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base]   require_explicit_package_overrides_for_builtin_materializations: false
2025-02-18T08:46:42.922008940Z 

[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] Notes -
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] * This is a temporary measure that will result in a deprecation warning, please ignore it for now. Elementary is working with the dbt-core team on a more permanent solution.
[2025-02-18, 08:46:42 UTC] {pod_manager.py:471} INFO - [base] * This message can be muted by setting the 'mute_ensure_materialization_override' var to true.
2025-02-18T08:46:42.922075379Z         

2025-02-18, 08:46:43 UTC] {pod_manager.py:471} INFO - [base] 08:46:42  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
[2025-02-18, 08:46:43 UTC] {pod_manager.py:471} INFO - [base] 08:46:42  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.29s]
[2025-02-18, 08:46:43 UTC] {pod_manager.py:471} INFO - [base] 08:46:43
[2025-02-18, 08:46:48 UTC] {pod_manager.py:471} INFO - [base] 08:46:43  1 of 1 START sql table model dbt_prod_marts.my_dbt_model  [RUN]
[2025-02-18, 08:46:48 UTC] {pod_manager.py:471} INFO - [base] 08:46:48  1 of 1 OK created sql table model dbt_prod_marts.my_dbt_model  [CREATE TABLE (20.0 rows, 2.0 GiB processed) in 5.43s]
[2025-02-18, 08:51:48 UTC] {pod_manager.py:489} INFO - [base] 08:46:48
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29  1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29  1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 1541.41s]
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29  Finished running 2 project hooks, 1 table model in 0 hours 25 minutes and 49.16 seconds (1549.16s).
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29  Completed successfully
[2025-02-18, 09:12:29 UTC] {pod_manager.py:471} INFO - [base] 09:12:29
[2025-02-18, 09:12:32 UTC] {pod_manager.py:489} INFO - [base] 09:12:29  Done. PASS=3 WARN=0 ERROR=0 SKIP=0 TOTAL=3
@Mika Kerman would appreciate any recommendations as this is currently blocking our production pipelines
a
@George Myrianthous are you running this on Databricks' All Purpose compute
g
Nope
@Mika Kerman Good morning; Have you got any updates on this?
n
Hey @George Myrianthous! We're looking into it.
@George Myrianthous What DWH are you using?
g
@Noya Offer BigQuery
👍 1
n
Ok, we'll need your dbt.log file to investigate some more 🙂
g
Hi @Noya Offer I will DM you the file in a second
🙌 2
f
Hi, did you get any clarity in what was causing this?
n
Hey @Fredrik Örn! In the latest versions, there’s a change to how columns are updated to 
dbt_columns
, which can impact on-run-end performance. Since this table isn't used in the Elementary UI, the best approach is to disable it by adding this to your `dbt_project.yml`:
Copy code
vars:
  columns_upload_strategy: none
This should help speed things up. Let me know if it works or if you need anything else!
f
Hello and thank you! drop happy Should that only affect elementary?
n
Yes 🙂 This only affects the 
dbt_columns
table in the Elementary schema.
33 Views
Open in Slack
PreviousNext
Powered by