<@U088WCP3B4P> elementary.all_columns_anomalies: ...
# ask-elementary-ai
y
@kapa.ai elementary.all_columns_anomalies: config: severity: warn arguments: ignore_small_changes: spike_failure_percent_threshold: 10 drop_failure_percent_threshold: 10 what does this test do? explain, and what it means when the test fails
k
A message from kapa.ai
This configuration is for the
elementary.all_columns_anomalies
anomaly-detection test with custom failure behavior:
Copy code
yaml elementary.all_columns_anomalies:   config:     severity: warn   arguments:     ignore_small_changes:       spike_failure_percent_threshold: 10       drop_failure_percent_threshold: 10
### What the test does -
elementary.all_columns_anomalies
runs anomaly detection on all columns of the table. - For each column, it monitors data-quality metrics (by type), such as: - Any type:
null_count
,
null_percent
- Strings:
min_length
,
max_length
,
average_length
,
missing_count
,
missing_percent
- Numerics:
min
,
max
,
average
,
zero_count
,
zero_percent
,
standard_deviation
,
variance
(and optionally
sum
). [[All columns](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/all-columns-anomalies)] - Elementary builds a historical baseline for these metrics and compares recent values (detection period) to past values (training period) to detect anomalies. [[How it works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works)] The
ignore_small_changes
block changes when an anomaly will actually cause the test to fail: -
spike_failure_percent_threshold: 10
– for spikes, the metric in the detection period must be more than 10% higher than the mean in the training period (and anomalous by z‑score) to fail. -
drop_failure_percent_threshold: 10
– for drops, the metric must be more than 10% lower than the mean in the training period (and anomalous by z‑score) to fail. [[Ignore small changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)] Also: -
config: severity: warn
means that when the test detects a failing anomaly, dbt will mark it as a warning, not a hard failure, so it won’t abort downstream builds. This uses dbt’s standard
severity
config, which community examples show working with
all_columns_anomalies
. [[Warn severity example](https://elementary-community.slack.com/archives/C02CTC89LAX/p1709141168413439)] ### What it means when the test “fails” For anomaly tests, “fail” means: - At least one monitored metric (e.g.,
null_percent
on some column) in the detection period is an outlier compared to its expected range from the training period, and - That change is large enough to pass your
ignore_small_changes
thresholds. [[How it works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works); [Ignore small changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)] So, in your configuration: - If an anomalous spike/drop is detected and it exceeds ±10% vs. the historical mean, the test result is: - “Warning” (because of
severity: warn
) – an anomaly was detected. - If no such anomaly is found (or changes are smaller than 10%), the test passes. Elementary explicitly defines: when a test fails, “it means that an anomaly was detected on this metric and dataset.” [[Test fails meaning](https://docs.elementary-data.com/data-tests/add-elementary-tests#what-does-it-mean-when-a-test-fails%3F)]
y
@kapa.ai - elementary.dimension_anomalies: config: severity: warn arguments: dimensions: - country_code ignore_small_changes: drop_failure_percent_threshold: 15 what about this one
k
A message from kapa.ai
This configuration defines a
dimension_anomalies
test with custom behavior:
Copy code
yaml - elementary.dimension_anomalies:     config:       severity: warn     arguments:       dimensions:         - country_code       ignore_small_changes:         drop_failure_percent_threshold: 15
### What this test does -
elementary.dimension_anomalies
counts rows grouped by the given dimensions and monitors how those counts change over time. [[Dimension anomalies intro](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies)] - Here,
dimensions: [country_code]
means Elementary tracks the row count per
country_code
, and looks for unexpected changes in the distribution of those counts. [[Dimension anomalies intro](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies); [dimensions param](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions)] - It compares recent counts (detection period) to historical counts (training period) using the anomaly-detection algorithm. [[How anomaly tests work](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works)] The
ignore_small_changes
block tightens when a detected anomaly will actually fail: -
drop_failure_percent_threshold: 15
means: - Even if a drop in row count for some
country_code
is statistically anomalous (high z-score), the test will only fail if that dropped value is more than 15% lower than the mean in the training period. [[ignore_small_changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)] (No spike threshold is set, so for spikes there is no additional percent filter beyond the anomaly score itself; the docs only say that when defined, both z‑score and the % drop/spike condition must hold to fail. [[ignore_small_changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)])
config: severity: warn
tells dbt to treat a failing anomaly as a warning rather than a hard error (this is standard dbt
severity
usage, shown in multiple Elementary examples). [[Add tests](https://docs.elementary-data.com/data-tests/add-elementary-tests)] ### What it means when this test fails According to the docs, when an anomaly test fails, “it means that an anomaly was detected on this metric and dataset.” [[What a failure means](https://docs.elementary-data.com/data-tests/add-elementary-tests#what-does-it-mean-when-a-test-fails%3F)] Concretely for your config, a failure means: - For at least one
country_code
, the row count per that country in the detection period: - Is an outlier compared with its expected range based on historical data (anomalous), and - Is more than 15% lower than the historical mean (a meaningful drop, not a small fluctuation). [[Dimension anomalies intro](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies); [ignore_small_changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes)] Because
severity: warn
is set, dbt will mark the test as a warning, but it still indicates a significant drop for at least one
country_code
.