https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> I’m building a custom JSON logger f...
# ask-elementary-ai
d
@kapa.ai I’m building a custom JSON logger for my dbt run and I need to capture all the rich data Elementary produces (everything that normally goes into 
elementary_test_results
and 
test_result_rows
, like anomaly scores, failed row samples, etc.). The issue is that my project's 
on-run-end
hook runs before the Elementary hook, so I can't query those tables yet—they haven't been updated with the current run's data. Is there a supported way to trigger my hook after Elementary finishes? Or is there a way to access that deep level of detail (like failed rows and Z-scores) directly from the 
results
object in memory? I tried using 
elementary.flatten_test(<http://result.node.to|result.node.to>_dict())
but it gives me only some static information.
k
A message from kapa.ai
You are correct that in dbt, package-level 
on-run-end
hooks (like those from Elementary) always run after your project's own 
on-run-end
hooks. There is currently no supported way in dbt to have your custom 
on-run-end
hook run after the Elementary hooks. This is a known limitation, and several users have encountered the same issue. The only workaround is to run your custom logic as a separate dbt operation after the main run, but this loses access to the in-memory 
results
context that is available during the hook execution. As one support staff put it: > "On a 
dbt
level, there is no way to do such a thing, package hooks will always run after the project hooks..." [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1710369355415909) Regarding accessing the deep level of detail (like failed rows, anomaly scores, etc.) directly from the 
results
object: the 
results
object available in the 
on-run-end
context contains the run results for each resource, but it does not include all the rich, post-processed data that Elementary writes to its tables. For example, failed row samples and anomaly scores are computed and stored by Elementary's own macros during its 
on-run-end
processing, and are not present in the raw 
results
object. This is confirmed by support staff: > "Elementary leverages the dbt 
results
jinja variable to get the results of all the resources who participated in an invocation. This variable is currently only accessible by dbt at the end of the DAG (in the context of an on-run-end hook)." [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1674071962667739) Additionally, the 
elementary.flatten_test()
macro only provides static test metadata, not the full enriched results or failed row samples. Summary: - There is no supported way to have your custom 
on-run-end
hook run after Elementary's hooks. - The 
results
object does not contain all the rich, post-processed data (like failed row samples or anomaly scores) that Elementary writes to its tables. - To access the full data, you must query the Elementary tables after the run is complete. If you need to automate this, consider running your custom logger as a separate dbt operation or script after the dbt run has fully completed and the Elementary tables have been updated. There is no built-in way to change the hook execution order or to access the fully enriched Elementary data in-memory during your own 
on-run-end
hook. If you have a specific use case, the Elementary team is open to feature requests—there was even discussion about possibly adding a callback macro that would run after Elementary's hooks, but this is not currently implemented or documented as a feature [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1666183249043369).
33 Views
Open in Slack
PreviousNext
Powered by