The issue you're experiencing is a common one: when running volume anomaly detection on a cumulative metric (like daily active users) with a filter for the current day (

activity_date = TIMESTAMP(CURRENT_DATE())

), each hourly run compares an incomplete, growing count for today against the completed count from previous days. This leads to false anomaly detections, as today's data is not yet complete. Best Practice Recommendation: - Configure the test to use completed time buckets only. Elementary's volume anomaly detection is designed to operate on completed time buckets (e.g., full days), not partial ones. By default, if you use a daily time bucket, the test will only evaluate the previous day's data, not the current (incomplete) day. This avoids the issue of comparing incomplete data to complete data and prevents false positives. - Remove the

where_expression

filtering for the current day. Instead, configure the test to use a

timestamp_column

(e.g.,

activity_date

) and set the

time_bucket

to daily. This way, Elementary will automatically group and compare data by full days, and only run anomaly detection on days that have finished accumulating data. Example configuration:

yaml - elementary.volume_anomalies:     timestamp_column: activity_date     time_bucket:       period: day       count: 1

- Run the test after the day is complete. If you want to be alerted as soon as possible after a day ends, schedule the test to run shortly after midnight, so it evaluates the previous day's complete data. If you need more immediate (intra-day) anomaly detection: - Consider using hourly buckets (

period: hour

) and ensure your training set is large enough to account for hourly fluctuations. However, this is only effective if your metric is meaningful at the hourly level and not just cumulative for the day. Otherwise, stick to daily buckets for cumulative metrics. References: - Elementary only runs anomaly detection on completed time buckets, so running the test during the day with a filter for the current day will always result in incomplete data and false alerts. The recommended approach is to let Elementary handle the time bucketing and avoid filtering for the current day in your test configuration [volume_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). - This behavior and the rationale are discussed in community support threads, where it's confirmed that the test should not be run on incomplete buckets and that the last bucket (today) is intentionally excluded to avoid this exact issue [Slack thread](https://slack.com/archives/C02CTC89LAX/p1706042023272269), [Slack thread](https://slack.com/archives/C02CTC89LAX/p1682044085432779). Summary: Remove the

where_expression

for the current day, use a

timestamp_column

and daily

time_bucket

, and let Elementary evaluate only completed days. This will prevent false anomaly detections due to incomplete data.

For the

training_period

, the default is 14 days, which means Elementary will use the previous 14 completed days to calculate the expected range for anomaly detection. You can adjust this value based on your data's seasonality and variability: - Increase

training_period

if you want the test to be less sensitive to outliers and reduce false positives. This is useful if your daily active user counts have a lot of natural fluctuation. - Decrease

training_period

if you want the test to be more sensitive and detect anomalies more quickly, but this may increase false positives if your data is noisy. Example configuration:

yaml training_period:   period: day   count: 14  # or another value that fits your data's behavior

If you use seasonality (e.g.,

seasonality: day_of_week

), Elementary will automatically extend the training period to ensure enough data points for each seasonality bucket (e.g., enough Mondays, Tuesdays, etc.) [training_period documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period). Summary: Start with the default 14 days, and adjust up or down based on how sensitive you want the anomaly detection to be and the natural variability in your data. If you use seasonality, be aware that the training period may be automatically extended to ensure robust anomaly detection for each seasonal group.