For truncate-and-load tables (where the table is fully replaced each load), the recommended configuration for Elementary’s

volume_anomalies

test is to avoid using a timestamp column. This ensures that each test execution counts the total row count of the table, rather than trying to bucket by a timestamp that resets with each load. This approach is specifically recommended for overwrite/truncate-and-load tables, as using a timestamp column can result in the test always seeing only the current day's data, which can cause issues with anomaly detection and expected ranges [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1710327533594939)]. ### Why did your test not fail on a large spike? - Expected range calculation: The expected range is based on the training data (previous runs). If your training set is small or includes large jumps, the expected range can become very wide, especially if the standard deviation is high. This can allow large spikes or drops to pass as "normal" [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1740060689781209)]. - ignore_small_changes: This setting only prevents the test from failing on small changes, but does not force a failure on large spikes if the expected range is wide due to high variance in the training data [[ignore_small_changes docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)]. - Training period and min_training_set_size: With a small training set (e.g., 3), a single large value can quickly shift the expected range upward, especially if the test includes the current value in the calculation [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1740060689781209)]. ### Recommendations for truncate-and-load tables: 1. Remove the timestamp_column from your test configuration. This will make the test use the total row count per run, which is the correct approach for truncate-and-load tables [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1710327533594939)]. 2. Increase the training period (e.g.,

min_training_set_size: 7

or more) to avoid the expected range being skewed by a single outlier. 3. Adjust anomaly_sensitivity: Lower sensitivity (higher value) makes the test less likely to fail; try a value of 3 or higher if you want to reduce false positives, or lower if you want to catch more anomalies. 4. Use ignore_small_changes to avoid noise, but note that it won’t force failures on large spikes if the expected range is wide. 5. fail_on_zero: true is good for catching empty loads. #### Example configuration for truncate-and-load table:

yaml - elementary.volume_anomalies:     min_training_set_size: 7     anomaly_sensitivity: 2     anomaly_direction: both     ignore_small_changes:       spike_failure_percent_threshold: 10       drop_failure_percent_threshold: 10     fail_on_zero: true     severity: warn     tags: ["elementary"]

Do not include a timestamp_column. ### Additional notes: - The expected range can include the current value, especially with small training sets, which can make the test less sensitive to sudden changes [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1740060689781209)]. - If you want the test to be more responsive, increase the training set size and/or adjust the sensitivity. - If you continue to see unexpected behavior, check the

data_monitoring_metrics

table to verify what values are being collected and used for training [[Anomaly troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)]. If you follow these recommendations and still see issues, please share your Elementary report or the relevant metrics for further troubleshooting.

Yes,

training_period

and

min_training_set_size

are different configuration options in Elementary anomaly detection tests. ### training_period - Purpose: Defines the time window (e.g., last 14 days) from which Elementary collects historical data to build the training set for anomaly detection. - How it works: For tests with a

timestamp_column

, Elementary looks back over the specified period (e.g., 14 days, 30 days) and uses those data points as the training set. The default is 14 days. - Effect: Increasing

training_period

makes the test less sensitive (more data, less likely to flag outliers); decreasing it makes the test more sensitive (less data, more likely to flag outliers) [[training_period docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period)]. ### min_training_set_size - Purpose: Sets the minimum number of data points required in the training set before the test will attempt to detect anomalies. - How it works: If there are fewer than

min_training_set_size

data points in the training set, the test will not run anomaly detection and will typically show a "not enough data to calculate anomaly" message. The default is 14 [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1682967620411319)]. ### Relationship and Verification - The training set is built from the data in the

training_period

, but if the number of data points is less than

min_training_set_size

, the test will not run. - You can check the actual number of data points used for the training set in the

metrics_anomaly_score

table, specifically in the

training_set

column. This column reflects how many previous data points were used for the current test, and it is directly affected by the

min_training_set_size

config [[Anomaly troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)]. Summary: -

training_period

= how far back in time to look for training data -

min_training_set_size

= minimum number of data points required to run the test - The

training_set

column in

metrics_anomaly_score

shows the actual count of training data points used, and is affected by

min_training_set_size

. If you set

min_training_set_size

to a value, the test will only run if at least that many data points are available in the training set, and you can verify this in the

training_set

column.

I'm sorry, but the knowledge sources do not explicitly explain why the

training_set_size

in the

metrics_anomaly_score

table can be less than the configured

min_training_set_size

. Based on the available documentation and Slack discussions, the expectation is that the test should not run anomaly detection unless the minimum training set size is met, and you should see a "not enough data to calculate anomaly" message if there are fewer data points than

min_training_set_size

[[Anomaly troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)]. If you are observing

training_set_size

values less than your configured minimum, this could indicate a bug or an edge case not covered in the documentation. In similar cases, users have been advised to check their package version, as some issues have been fixed in more recent releases [[Slack discussion](https://slack.com/archives/C02CTC89LAX/p1692691883697119)]. If this behavior persists, I recommend sharing your configuration and findings with the Elementary team, as it may require further investigation.