Hi folks, i am facing a long time during on-run-end hook after running dbt run/test, it takes about 20min+. Will anyone be able to help?

This could be the result of the same thing that's causing some DBs on run end hooks to fail with something like the following

maximum number of expressions in a list exceeded, expected at most 16,384, got 17,861

Might be that your DB does not have a hard limit but its still taking longer and longer (It seems to incrementally get worse) the elementary folk are being very quiet about it considering the impact. What DB do you use out of curiosity? Big Query and Snowflake definitely have the insert limit on whatever query the on run end hook is performing. https://elementary-community.slack.com/archives/C02CTC89LAX/p1706636103858569?thread_ts=1706626800.346159&cid=C02CTC89LAX

I am using BQ

Could maybe expect to see the list exceeded error - this is a potential expected behavior from what I can see in the upload_run_results macro. One of the post's i reference was from 3 months ago and has some recent chatter (7 days ago) - they rolled back to previous versions where this did not happen.

Hi @Kelvin, ill be glad to help. I need some information tho: • which dbt version are you using? • which elementary dbt package are you using? • what command are you running exactly? • does it happen all the time, or in specific instances? if so, when does it happen and when does it not?

Hi @Ofek Weiss thanks for helping • dbt-core==1.7.6 • elementary version: 0.12.1 • It happens for ◦ edr run-operation upload-source-freshness --project-dir . ◦ dbt run/dbt test elementary hooks ◦ edr monitor … • Pretty much all the time

@Kelvin Brangissko can you attach

edr.log

so we can investigate further? If possible, please open an Github issue with all the info above so we can track and make sure to solve it

Sorry folks, i didn’t see this message. Looping in my teammate to help @Bhavya Kalra