<@U088WCP3B4P> I want to get started with freshnes...
# ask-elementary-ai
n
@kapa.ai I want to get started with freshness and volumes test do you have links for me?
k
A message from kapa.ai
Absolutely! To get started with freshness and volume tests in Elementary, here are some helpful links and resources: - Freshness Anomalies Test: This test monitors the freshness of your table over time, detecting delays in data updates. You can find configuration details and examples here: [Freshness Anomalies Test Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies). - Volume Anomalies Test: This test monitors the row count of your table over time to detect unexpected drops or spikes. Configuration and usage details are available here: [Volume Anomalies Test Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). - How to Add Elementary Tests: For step-by-step instructions on adding these anomaly detection tests to your dbt project, see: [Add Anomaly Detection Tests](https://docs.elementary-data.com/data-tests/add-elementary-tests). - Best Practices and Use Cases: For recommendations on when and how to use these tests, check out: [Detection and Coverage Best Practices](https://docs.elementary-data.com/best-practices/detection-and-coverage#common-testing-use-cases). If you’re using Elementary Cloud, you also get out-of-the-box automated freshness and volume monitors with zero configuration. Learn more here: [Automated Freshness & Volume Monitors](https://docs.elementary-data.com/features/anomaly-detection/automated-monitors). These resources should help you get started quickly and effectively!
n
@kapa.ai I have the following tests for volume and freshness: tests: - elementary.volume_anomalies: timestamp_column: CREATED_TIME anomaly_direction: drop # It will check the volumes changes based on created time. Default is the time bucket 1 day - elementary.freshness_anomalies: timestamp_column: CREATED_TIME However when I look in DB the column "Column_name "of the metrics_anomaly_score is empty and in the report I dont see the expected results. Do I miss something?
k
A message from kapa.ai
Based on the information in the knowledge sources, here are some possible reasons why the
column_name
in the
metrics_anomaly_score
table is empty and why you might not see the expected results in your report: 1. Volume and Freshness Tests Are Table-Level The
elementary.volume_anomalies
and
elementary.freshness_anomalies
tests are table-level tests, not column-level. For these tests, the metrics are associated with the table as a whole, so the
column_name
field in the metrics tables (including
metrics_anomaly_score
) will be empty or null. This is expected behavior for these types of tests. Only column-level anomaly tests (like
elementary.column_anomalies
) will populate the
column_name
field with a value [see this discussion](https://slack.com/archives/C02CTC89LAX/p1708533386888479). 2. Not Enough Training Data If you see messages like "Not enough data to calculate anomaly score" or if the report is missing expected results, it could be due to insufficient training data. By default, Elementary requires a minimum number of historical data points (typically 14) to calculate anomaly scores. If your table does not have enough historical data in the
CREATED_TIME
column, or if the test hasn't run enough times to collect sufficient data, the anomaly score will not be calculated [see troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). 3. Timestamp Column Issues If the
CREATED_TIME
column has missing, null, or incorrectly formatted values, metrics may not be collected as expected. Ensure that
CREATED_TIME
is populated and in the correct format for all relevant rows [see troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). 4. Test Configuration and Execution Make sure that: - The tests are configured correctly in your YAML. - You are running
dbt test
and the Elementary on-run-end hooks are completing successfully. - The Elementary tables in your warehouse are up to date and compatible with your package version [see this answer](https://slack.com/archives/C02CTC89LAX/p1688995519653729). Summary: For table-level anomaly tests, it is normal for
column_name
to be empty in
metrics_anomaly_score
. If you are not seeing results, check for sufficient training data, correct timestamp column values, and successful test execution. If you want to monitor anomalies at the column level, use
elementary.column_anomalies
or
elementary.all_columns_anomalies
tests. If you have confirmed all the above and still do not see results, please provide more details about your data and test run history for further troubleshooting.
n
Hey @Niels van Beem! Was this answer helpful?