https://linen.dev logo
Join Slack
Powered by
The elementary on-run-end hook for our production ...
# community-support
s
The elementary on-run-end hook for our production job sometimes takes 10+ minutes to start after finishing all the tests. Since we're on Databricks, that means 10+ minutes of compute we are paying for unnecessarily (unless I'm missing something). For example, see the dbt logs in the thread below. Am I misinterpreting the logs? If not, is there any way to reduce this lag time?
Copy code
14:22:28 1053 of 1139 PASS <final test name>  [PASS in 65.22s]
14:22:28  
14:22:28
14:22:28  Running 1 on-run-end hook
14:33:04 1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN]
14:33:04 1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]
note that the last test completes at 2:22 PM UTC but the 
1 of 1 START hook: elementary.on-run-end.0
line isn't written until 11 minutes later at 2:33 PM UTC
The only thing the SQL warehouse in question is used for is this production job, so concurrency issues can't be the cause of the delay
also note that the final test to complete is 
1053 of 1139
because it takes longer than any of the remaining 86 tests, not because this is an incomplete log file
d
Hey @Samuel Garfield, here you can see the dbt code that runs the hooks. 1. In line 353 it's the event that writes the 
Running 1 on-run-end hook
log 2. In line 369, it's the event that writes the 
1 of 1 START hook: elementary.on-run-end.0
Between those lines, in 362, you can see - 
sql = self.get_hook_sql(...)
That's what takes time. It's compiling our end hook, and during the compilation we're doing all of our logic. So the logs are a bit deceiving.
🙏 1
👍 1
31 Views
Open in Slack
PreviousNext
Powered by