is there backoff handling included in the

on-run-end

hooks? we're consistently running into concurrent transaction exceptions on the

dbt_elementary

tables because many of our data pipelines are finishing at the same time daily

Hey Ian! What DWH are you using? We're writing a best practices guide for this use case, will be published on Monday. I'll keep you updated! 🙂

hi Noya! we're running in BigQuery

Hey Ian, We just added some instructions for your case 🙂 Let me know if this helps!

thank you for sharing! unfortunately this doesn't totally apply to our use case, the jobs that are raising the exception are daily jobs (not concurrent executions of the same job). we load data from ~50 different sources into our warehouse every morning and the issue we're having occurs when these pipelines are finishing around the same time

Thanks for sharing and great to hear! Just to understand for my own curiosity, it was solved by optimizing those long running models?

we ended up just removing the post-hooks from the long running models, they go directly into our stakeholders' BQ projects so we don't need to track their performance