<@U088WCP3B4P> In elementary schema data_monitorin...
# ask-elementary-ai
n
@kapa.ai In elementary schema data_monitoring_metrics table column bucket_start has null value, but bucket_end has correct values. My volume anomaly test is for a truncate and load table, so there is no time_stamp column configured. Why bucket_start column is empty?
k
A message from kapa.ai
When you configure a volume anomaly test in Elementary without specifying a timestamp column (which is common for truncate-and-load tables), the test does not create time buckets based on data timestamps. Instead, each test run calculates the metric (e.g., row count) for the entire table at the time of the run. In this scenario: - The bucket_start column in the data_monitoring_metrics table will be null because there is no timestamp to define the start of a time bucket. - The bucket_end column will contain the timestamp of when the test was run, effectively marking the end of the "bucket" (which, in this case, is just the test execution time). This is expected behavior for tables without a timestamp column. The anomaly detection relies on metrics collected from each test run, not from historical time buckets within the data itself. You can see this explained in several support responses and documentation: > "When using anomaly test without a timestamp column... The test will simply take 'snapshot' of the current row count and compare it to previous snapshots of the time of running dbt test." [Slack Support Response](https://slack.com/archives/C02CTC89LAX/p1695060702152949) > "The BUCKET_END contains the test run time. All 16 records are of the same model? ... This is how it should be for tests with no timestamp column." [Slack Support Response](https://slack.com/archives/C02CTC89LAX/p1687786556886149) So, the null values in bucket_start are normal and indicate that the test is operating in "snapshot" mode, not time-bucketed mode.
n
@kapa.ai But then when metrics_anomaly_score table model runs in elementary to calculate training_avg, training_stddev, training_set_size The logic uses (partition by metric_name, full_table_name, column_name order by bucket_start asc rows between unbounded preceding and current row) And because bucket_start is null in data_monitoring_metrics table. we get inconsistent and wrong values in metrics_anomaly_score table.
k
A message from kapa.ai
You are correct in your observation, and this issue has been identified by the Elementary community as well. When there is no timestamp column configured, the bucket_start column in data_monitoring_metrics remains null. However, the metrics_anomaly_score model uses window functions that partition and order by bucket_start, which leads to incorrect calculations when bucket_start is always null. This problem is described in an open GitHub issue: > In metrics_anomaly_score.sql, the metric_value is not partitioned by dimensions when using the dimension properties. This affects the calculation of the anomaly score since it is derived from the average of the metric values across ALL dimensions. > ...the metric_value is not being partitioned by dimensions: > avg(metric_value) over (partition by metric_name, full_table_name, column_name order by bucket_start asc rows between unbounded preceding and current row) as training_avg > ...The average metric_value should be partitioned by dimension_value when dimensions are being used > ...I think we would also want to include dimension in the partition, as well. > [GitHub Issue: Partition by dimension in metrics_anomaly_score](https://github.com/elementary-data/elementary/issues/1729) While the issue above focuses on partitioning by dimension, the underlying problem is the same: when bucket_start is null (as in your case), the window function does not behave as intended, leading to inconsistent or incorrect training_avg, training_stddev, and training_set_size values in metrics_anomaly_score. This is a known limitation when running anomaly detection tests without a timestamp column. The Elementary team is aware of this and there are discussions about improving the logic to handle such cases more robustly. Summary: - When no timestamp column is configured, bucket_start is null. - The current logic in metrics_anomaly_score that orders by bucket_start does not work correctly when all values are null. - This leads to incorrect anomaly calculations, as you described. - The issue is recognized by the Elementary community and is being tracked for improvement. If this is blocking your workflow, you may want to follow or contribute to the ongoing discussion in the referenced GitHub issue.
n
@kapa.ai Which elementary table is used to calculated the expected range we see in the edr report? And which table is used to finally decided if current test is anomaly?
k
A message from kapa.ai
The expected range you see in the EDR report is calculated based on historical metrics stored in the
data_monitoring_metrics
table. This table accumulates metric values (such as row count, freshness, etc.) over time for each anomaly detection test. The anomaly detection logic uses these historical values to compute the expected range for the monitored metric, typically using statistical measures like average and standard deviation over a training period [Dbt package models - Anomaly detection](https://docs.elementary-data.com/dbt/package-models#anomaly-detection). To determine if the current test result is an anomaly, Elementary uses the
metrics_anomaly_score
view. This view runs the anomaly detection logic, comparing the latest metric value to the expected range (derived from
data_monitoring_metrics
) and calculating the anomaly score. If the score exceeds the configured threshold, the result is flagged as an anomaly [Dbt package models - Anomaly detection](https://docs.elementary-data.com/dbt/package-models#anomaly-detection). In summary: - Expected range calculation:
data_monitoring_metrics
- Anomaly decision:
metrics_anomaly_score
view This is also supported by community responses, which confirm that
data_monitoring_metrics
holds the historical metrics and
metrics_anomaly_score
provides visibility into anomaly detection results [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1666376716952649).