Elementary’s anomaly detection for the

column_anomalies

test does not use a rolling average of previous max values as the baseline. Instead, it calculates the baseline (expected value) for the monitored metric (e.g.,

max(total_cost)

) using the average and standard deviation from the training period, which by default includes recent historical buckets (e.g., days). The threshold for flagging a spike is then determined by both the anomaly sensitivity (number of standard deviations) and the

spike_failure_percent_threshold

if you set

ignore_small_changes

. Specifically, the upper threshold for a spike is calculated as the greater of: -

training_avg + anomaly_sensitivity * training_stddev

-

(1 + spike_failure_percent_threshold/100) * training_avg

So, if your training average is 28.19 and you set

spike_failure_percent_threshold: 30

, the threshold would be at least 36.64, but it could be higher if the standard deviation is large and the first formula yields a higher value. The test will only alert if the observed value exceeds this computed threshold. Additionally, Elementary includes the value from the detection period (the value being tested) in the calculation of the training average and standard deviation. This can "smooth" the baseline upwards, especially if the spike is significant, making it less likely to be flagged as an anomaly. This behavior has been noted as a source of confusion and is being discussed as a potential issue to address in future releases, as it can cause the test to miss anomalies that would otherwise be detected if the detection period value were excluded from the training set [[Anomaly detection. Detection period value is included in the calculation of the AVG value for the training period](https://github.com/elementary-data/elementary/issues/1560)]. So, in your case, if the spike to 40 is included in the training period, it will raise the average and possibly the standard deviation, increasing the threshold and potentially preventing the alert. Earlier values like 24.18 and 28.19 contribute to the average, but a large spike in the detection period can have a disproportionate effect due to this inclusion. Summary: - The baseline is the average (and stddev) of the training period, not just the previous max. - The detection period value is included in the training calculation, which can raise the threshold. - The threshold is the greater of the sensitivity-based value or the percent threshold. - Smoothing from both earlier values and inclusion of the spike itself can prevent an alert, even if the spike exceeds your expected threshold [[Anomaly detection. Detection period value is included in the calculation of the AVG value for the training period](https://github.com/elementary-data/elementary/issues/1560)]. If you want more sensitive detection, consider reducing the training period or anomaly sensitivity, but be aware of the current behavior regarding the inclusion of the detection period value.

Great questions! Here’s what the knowledge sources say about your concerns: 1. Is there a recommended way to exclude the detection period value from training, or is this behavior fixed for now? Currently, there is no built-in Elementary parameter to exclude the detection period value from the training set. This behavior—where the detection period is included in the training period—has been acknowledged as a known issue and is under discussion in the community and GitHub issues. Some users have tried workarounds using the

anomaly_exclude_metrics

parameter, but this will exclude the value from both the detection and training periods, which is not the intended effect. Trusted support staff have confirmed there is no official workaround at this time [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1740069258974599)]. 2. Is it safe to assume that shortening the

training_period

(e.g., from 14 days to 5) might reduce smoothing and make the test more sensitive? Yes, reducing the

training_period

will make the test more sensitive. A shorter training period means the average and standard deviation are calculated from fewer data points, so outliers (like your spike) will have a greater impact, and the expected range will be narrower. This increases the likelihood of detecting anomalies, but also raises the risk of false positives [[training_period docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period)]. 3. Are there any updates on the potential change to this logic — e.g., skipping the detection value from the training window — in future versions? As of now, this behavior is still present and tracked in open GitHub issues. The Elementary team has acknowledged the concern and is open to changing the default or making it configurable, but there is no official release or fix yet. You can follow progress and discussion in these issues: - [Exclude detection period from training period?](https://github.com/elementary-data/elementary/issues/1491) - [Detection period value is included in the calculation of the AVG value for the training period](https://github.com/elementary-data/elementary/issues/1560) Trusted support staff have stated, “there’s no workaround for this with the existing Elementary parameters. We will look into this some more to find a solution” [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1740069258974599)]. Summary: - There is no current way to exclude the detection value from training via configuration. - Shortening the training period will make the test more sensitive. - The Elementary team is aware of the issue, but no fix or new option has been released yet. Let me know if you need more details or want to discuss alternative anomaly detection strategies!

To determine if the spike to 40 is included in the training period, you should review the data used for the training set in your anomaly test. According to Elementary’s documentation and support responses, the training period typically includes the detection period value (i.e., the value being tested) by default. This means that unless a future version changes this logic, the spike to 40 would be part of the training set used to calculate the average and standard deviation for anomaly detection [[GitHub Issue #1491](https://github.com/elementary-data/elementary/issues/1491)]. To confirm this in your environment, you can: - Query the underlying metrics tables (such as

data_monitoring_metrics

or the temporary tables created for your test) to see which values are included in the training set for the relevant detection period. - Review the SQL logic in the Elementary package, which currently does not exclude the detection value from the training window [[GitHub Issue #1560](https://github.com/elementary-data/elementary/issues/1560)]. - Check the anomaly report or the metrics tables for the specific time bucket (e.g., the day with the spike) and see if that value appears in both the training and detection calculations. If you see the spike value present in the calculation of the training average or standard deviation for the same period it is being tested, then it is included in the training period. This is the current default behavior in Elementary [[Slack thread](https://slack.com/archives/C02CTC89LAX/p1740069258974599)].