<@U088WCP3B4P>, we use databricks and source fresh...
# ask-elementary-ai
o
@kapa.ai, we use databricks and source freshness. I also have this flag in the dbt_project.yml source_freshness_run_project_hooks: true # elementary setting for uploading source freshness results. And yet, I don't get alerts when the source is stale. I see in the workflows that those elementary hooks did not run.
k
A message from kapa.ai
Based on the knowledge sources, here are some key points to help debug your issue: 1. dbt Version Requirement: The flag
source_freshness_run_project_hooks: true
in your
dbt_project.yml
is only effective for dbt version 1.8 and above. This flag enables Elementary to automatically collect source freshness results using on-run-end hooks, which is required for automatic alerting on source freshness failures. If you are using a version lower than 1.8, the hooks will not run, and you must use the manual upload method instead ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 2. Elementary Hooks Not Running: If you are on dbt 1.8+ and still do not see the hooks running, double-check: - That the flag is correctly indented and placed in your
dbt_project.yml
. - That your workflow is not overriding or skipping project hooks. - That you are running
dbt source freshness
in an environment where the hooks are enabled and not being suppressed by any CLI flags or environment settings. 3. Manual Upload (for dbt <1.8 or troubleshooting): If the hooks are not running, you can still upload source freshness results manually: 1. Run
dbt source freshness
2. Then run
edr run-operation upload-source-freshness --project-dir <path to your project>
3. Finally, run
edr monitor
to trigger alerts ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 4. Databricks Compatibility: There is no explicit mention in the knowledge sources of Databricks-specific issues with source freshness hooks, so the general guidance above applies. 5. UI and Alerts: If source freshness results are collected, they should be presented in the UI and in alerts upon failure. If you are not seeing alerts, ensure that the results are actually being uploaded and that your alerting configuration (e.g., Slack channel, alert severity) is correct ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). Summary: - Ensure you are on dbt 1.8+ for the flag to work. - Double-check your
dbt_project.yml
and workflow setup. - If hooks are not running, use the manual upload method. - Confirm that your alerting configuration is correct. If after these checks the hooks still do not run, there may be an issue with your dbt or Elementary setup that is not covered in the current knowledge sources. In that case, please provide more details about your dbt version, Elementary version, and workflow setup for further troubleshooting. References: - [Collect dbt source freshness results (OSS Guide)](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness) - [Cloud Guide: Collect dbt source freshness results](https://docs.elementary-data.com/cloud/guides/collect-source-freshness) - [Github Issue: dbt-core 1.8 support for source freshness hooks](https://github.com/elementary-data/elementary/issues/1425)
o
I use dbt 1.9.1 @kapa.ai and the config is correctly indented.
k
A message from kapa.ai
Since you are using dbt 1.9.1 and have correctly set the flag
source_freshness_run_project_hooks: true
in your
dbt_project.yml
, Elementary should be able to automatically collect source freshness results and trigger alerts when a source is stale. This is the intended behavior for dbt versions 1.8 and above, as documented in the official guides ([Collect dbt source freshness results](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). However, you mentioned that the Elementary hooks are not running and you are not receiving alerts. Based on the knowledge sources, here are the most relevant troubleshooting steps and considerations: 1. Flag Placement and Indentation: You confirmed the flag is correctly indented. This is important, as incorrect placement can prevent hooks from running. 2. Elementary Hooks Not Running: If the hooks are not running, it could indicate an issue with how dbt is executing your project or how your workflow is set up. The knowledge sources do not provide a specific troubleshooting checklist for this scenario, but they do confirm that with dbt 1.8+ and the flag set, hooks should run and source freshness results should be collected automatically ([Cloud Guide](https://docs.elementary-data.com/cloud/guides/collect-source-freshness)). 3. Manual Upload as a Workaround: If automatic collection is not working, you can still use the manual upload method as a workaround:
Copy code
dbt source freshness    edr run-operation upload-source-freshness --project-dir <path to your project>    edr monitor
This sequence is confirmed to work for collecting and alerting on source freshness results ([OSS Guide](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 4. No Known Databricks-Specific Issues: The knowledge sources do not mention any Databricks-specific limitations or issues with source freshness collection. 5. If Hooks Still Don’t Run: If you have verified the flag and dbt version, and hooks still do not run, the knowledge sources do not provide further troubleshooting steps for this specific scenario. It may be necessary to reach out to the Elementary team for more targeted support, as suggested in the documentation. Summary: - With dbt 1.9.1 and the correct flag, Elementary should collect source freshness results and alert on failures. - If hooks are not running, use the manual upload method as a workaround. - The knowledge sources do not provide additional troubleshooting steps for hooks not running in this scenario. If you need further help, consider sharing logs or reaching out to the Elementary team as recommended in the documentation. [Collect dbt source freshness results](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)
o
all good, it was an issue with target when running dbt source freshness