Ondra Paul
06/02/2025, 11:27 AMkapa.ai
06/02/2025, 11:27 AMkapa.ai
06/02/2025, 11:27 AMsource_freshness_run_project_hooks: true in your dbt_project.yml is only effective for dbt version 1.8 and above. This flag enables Elementary to automatically collect source freshness results using on-run-end hooks, which is required for automatic alerting on source freshness failures. If you are using a version lower than 1.8, the hooks will not run, and you must use the manual upload method instead ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 2. Elementary Hooks Not Running: If you are on dbt 1.8+ and still do not see the hooks running, double-check: - That the flag is correctly indented and placed in your dbt_project.yml. - That your workflow is not overriding or skipping project hooks. - That you are running dbt source freshness in an environment where the hooks are enabled and not being suppressed by any CLI flags or environment settings. 3. Manual Upload (for dbt <1.8 or troubleshooting): If the hooks are not running, you can still upload source freshness results manually: 1. Run dbt source freshness 2. Then run edr run-operation upload-source-freshness --project-dir <path to your project> 3. Finally, run edr monitor to trigger alerts ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 4. Databricks Compatibility: There is no explicit mention in the knowledge sources of Databricks-specific issues with source freshness hooks, so the general guidance above applies. 5. UI and Alerts: If source freshness results are collected, they should be presented in the UI and in alerts upon failure. If you are not seeing alerts, ensure that the results are actually being uploaded and that your alerting configuration (e.g., Slack channel, alert severity) is correct ([docs](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). Summary: - Ensure you are on dbt 1.8+ for the flag to work. - Double-check your dbt_project.yml and workflow setup. - If hooks are not running, use the manual upload method. - Confirm that your alerting configuration is correct. If after these checks the hooks still do not run, there may be an issue with your dbt or Elementary setup that is not covered in the current knowledge sources. In that case, please provide more details about your dbt version, Elementary version, and workflow setup for further troubleshooting. References: - [Collect dbt source freshness results (OSS Guide)](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness) - [Cloud Guide: Collect dbt source freshness results](https://docs.elementary-data.com/cloud/guides/collect-source-freshness) - [Github Issue: dbt-core 1.8 support for source freshness hooks](https://github.com/elementary-data/elementary/issues/1425)Ondra Paul
06/02/2025, 11:27 AMkapa.ai
06/02/2025, 11:29 AMkapa.ai
06/02/2025, 11:29 AMsource_freshness_run_project_hooks: true in your dbt_project.yml, Elementary should be able to automatically collect source freshness results and trigger alerts when a source is stale. This is the intended behavior for dbt versions 1.8 and above, as documented in the official guides ([Collect dbt source freshness results](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). However, you mentioned that the Elementary hooks are not running and you are not receiving alerts. Based on the knowledge sources, here are the most relevant troubleshooting steps and considerations: 1. Flag Placement and Indentation: You confirmed the flag is correctly indented. This is important, as incorrect placement can prevent hooks from running. 2. Elementary Hooks Not Running: If the hooks are not running, it could indicate an issue with how dbt is executing your project or how your workflow is set up. The knowledge sources do not provide a specific troubleshooting checklist for this scenario, but they do confirm that with dbt 1.8+ and the flag set, hooks should run and source freshness results should be collected automatically ([Cloud Guide](https://docs.elementary-data.com/cloud/guides/collect-source-freshness)). 3. Manual Upload as a Workaround: If automatic collection is not working, you can still use the manual upload method as a workaround: dbt source freshness edr run-operation upload-source-freshness --project-dir <path to your project> edr monitor This sequence is confirmed to work for collecting and alerting on source freshness results ([OSS Guide](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)). 4. No Known Databricks-Specific Issues: The knowledge sources do not mention any Databricks-specific limitations or issues with source freshness collection. 5. If Hooks Still Don’t Run: If you have verified the flag and dbt version, and hooks still do not run, the knowledge sources do not provide further troubleshooting steps for this specific scenario. It may be necessary to reach out to the Elementary team for more targeted support, as suggested in the documentation. Summary: - With dbt 1.9.1 and the correct flag, Elementary should collect source freshness results and alert on failures. - If hooks are not running, use the manual upload method as a workaround. - The knowledge sources do not provide additional troubleshooting steps for hooks not running in this scenario. If you need further help, consider sharing logs or reaching out to the Elementary team as recommended in the documentation. [Collect dbt source freshness results](https://docs.elementary-data.com/oss/guides/collect-dbt-source-freshness)Ondra Paul
06/02/2025, 12:14 PM