https://linen.dev logo
Join Slack
Powered by
Hi all! We are happy users of Elementary but there...
# product-feedback
v
Hi all! We are happy users of Elementary but there is one design decision that we keep bumping into: the fact that everything is uploaded using 
on-run-end
hook, i.e., only after the whole dbt run has finished. Are there any plans of changing that in the not too distant future, i.e., update the internal tables during the run? (Sorry if this topic is already covered somewhere — searched but didn’t manage to find here nor in Github issues. Background in the thread 🧵)
Our dbt run takes quite long and we’d like to be notified as soon as a model has failed (so we could fix asap, preferably before our automatic retry kicks in). We also want to allow downstream outside dbt to continue as soon as possible which we could easily implement on top of elementary tables using SQL sensors in Airflow if elementary tables were updated as the models run (rather than at the end). As it stands we’ve implemented that logging to internal tables and alerting ourselves using dbt callbacks and configured SQL sensors in Airflow on top of those — it works fine but it’s a pity that we cannot build on top of Elementary just due to this design decision (because finding out that something has failed up to a few hours later is not good enough for us). I imagine this topic to be relevant also for the recently launched incident management features because such delays in alerting could mean significantly increased time to resolution.
o
Hi @Valdar Tammik! that's really interesting! the reason why we rely on on-run-end is because that's the only way we can get the run results from dbt as a dbt package (they expose the results object as a jinja variable only at the end of the run). Relying on callbacks is only available if we can wrap the dbt CLI with a python code (for example our CLI). This is more complicated to deploy as you will need to deploy this and run it instead of dbt, and it will make it available to less users as we have more users installing and using the dbt package than users running the CLI and for them it won't be possible to use this mode. That said, I'm happy to discuss this with the team / hear ideas on other approaches to this
Also btw - do you use dbt cloud or dbt core? I assume core?
v
Wouldn't have thought many people are using only the elementary dbt package (without the CLI). In any case, perhaps the CLI could rely on the structured logs so it could be independent of the dbt CLI (as long as it can access the logs written by it). This way it could be run in different ways (either in parallel as we'd like to or after) and there'd be no need for the dbt package for logging if the CLI is used for that (but it doesn't need to replace it; would just provide an alternative way to populate the tables)? We're using dbt core, yes.
e
Hey folks, we would also be interested in a slightly different way in handling the 
on-run-end
hook is used. We have roughly 1000 dbt models that are run every day via Airflow. We would love to abstract out the uploading of the run results to a separate task to allow downstream models to begin running sooner, and also bypass/handle any issues with the 
Serializable isolation violation
that we occasionally run into. Can you point me to where the jinja variable is exposed? It may be possible for us to dump that into temporary storage in S3 and then have Elementary receive these independently
33 Views
Open in Slack
PreviousNext
Powered by