<@U088WCP3B4P> having an issue with column anomali...
# ask-elementary-ai
o
@kapa.ai having an issue with column anomalies which isn't triggering an alert: I'm testing a difference in the amount of zeros using test data in the last 2 days in a column and i've made the entire column 1s in the last 2 days using transformations and it still doesn't seem to trigger an error? Not sure what I am doing wrong:
Copy code
- name: has_been_subscribed
  data_tests:
    - elementary.column_anomalies:
        column_anomalies: zero_count
        timestamp_column: created_timestamp
        anomaly_sensitivity: 1
        time_bucket:
          period: day
          count: 1
        training_period:
          period: day
          count: 14
        detection_period:
          period: day
          count: 2
then i've added this below to make sure that there is a disproportionate amount of zeros in the last 2 days. I've checked the data on the DWH and it is exactly how I expect it to look:
Copy code
, case
        when
            {{ move_date_forward('exploded_customers.created_timestamp') }} >=
            timestamp_sub(current_timestamp(), interval 2 day)
            then 1
        else exploded_customers.has_been_subscribed
    end as has_been_subscribed
In the last 14 days there is about a 2:1 ratio of has_been_subscribed being 0 or 1. So in the last 2 days (the detection period) i've set the column to be 1 all the time but no test failures happen.
k
A message from kapa.ai
Based on your configuration and the behavior you described, here are some key points and troubleshooting steps that may explain why the
elementary.column_anomalies
test is not triggering an alert for the change in zero counts: ### 1. Training and Detection Periods - Your test is configured with a
training_period
of 14 days and a
detection_period
of 2 days. This means the test compares the last 2 days (detection period) to the previous 14 days (training period) using daily buckets. - If the last 2 days have all 1s (no zeros), and the previous 14 days have a 2:1 ratio of 0s to 1s, a significant drop in zero count should be detected as an anomaly, especially with
anomaly_sensitivity: 1
(which is quite sensitive). ### 2. Sensitivity and Thresholds - The default sensitivity is 3, but you set it to 1, which should make the test more likely to flag deviations. However, if the standard deviation in the training period is high (due to variability in the zero count), the expected range may still be wide, and the absence of zeros in the detection period might not be considered anomalous enough. This is supported by a similar case where 0 values were not flagged as anomalies due to high variance in the training set, and only a lower sensitivity (1.5 or under) would catch them. You already set sensitivity to 1, so this should not be the issue unless the variance is still too high for the test to trigger [Slack thread](https://slack.com/archives/C02CTC89LAX/p1736937571030059). ### 3. Data Collection and Bucketing - Ensure that your
created_timestamp
column is populated and in the correct format, as Elementary relies on this for bucketing. If there are gaps or issues with the timestamp, buckets may not be created as expected, and the test may not evaluate the correct periods [Anomaly Troubleshooting Guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). - You can check the
data_monitoring_metrics
table to verify that metrics are being collected for each bucket as expected. ### 4. Metric Calculation - The test uses the zero count per bucket (day) for the last 2 days and compares it to the distribution in the previous 14 days. If the training period has high variability, the absence of zeros in the detection period may not be statistically significant enough to trigger an alert, even with a low sensitivity. ### 5. Additional Checks - Confirm that the test is running after the data transformation is applied and that the data in the warehouse matches your expectations for the last 2 days. - Check the Elementary report or the
metrics_anomaly_score
table to see the calculated z-scores and thresholds for the zero count metric. This can help you understand why the test is not failing [Anomaly Troubleshooting Guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). ### 6. Consider Using
fail_on_zero
- If your main concern is the presence or absence of zeros, you might consider using the
fail_on_zero
parameter, which will fail the test if a zero is detected (or not detected, depending on your use case) [Anomaly Params](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-params). --- Summary: The most likely reason the test is not failing is due to the statistical properties of your training set (high variance in zero counts), which makes the absence of zeros in the detection period not statistically significant enough to trigger an anomaly, even with a low sensitivity. Check the actual metrics and z-scores in the Elementary tables to confirm this. If you need a strict rule about zeros, consider using a custom test or the
fail_on_zero
parameter. If you need more targeted troubleshooting, check the
data_monitoring_metrics
and
metrics_anomaly_score
tables for your model and column to see exactly what values are being compared and what thresholds are being used. This will clarify why the test is not failing as expected.
o
@kapa.ai the
created_timestamp
should be working as it works to make time buckets for other tests like
volume_anomalies
k
A message from kapa.ai
Thank you for clarifying that
created_timestamp
is working for other tests like
volume_anomalies
. This suggests that the timestamp column itself is not the issue, and time bucketing should be functioning correctly for your
column_anomalies
test as well. Given this, the most likely reasons your
zero_count
anomaly test is not triggering an alert are: 1. Statistical Variance in Training Data: If your training period (the previous 14 days) has a high variance in the zero count, the anomaly detection threshold may be wide. Even with
anomaly_sensitivity: 1
, if the zero count fluctuated a lot in the training period, the absence of zeros in the last 2 days might not be considered statistically significant enough to trigger an alert. This behavior is consistent with how Elementary calculates anomalies, as confirmed in community discussions: > "The 0 values would only be considered an anomaly when the anomaly_sensitivity configuration will be set to 1.5 or under (the default sensitivity is 3)." [Slack thread](https://slack.com/archives/C02CTC89LAX/p1736937571030059) 2. Check Data Collection: Since your timestamp column is working for other tests, you can verify that metrics are being collected for each bucket in the
data_monitoring_metrics
table. This will confirm that the test is evaluating the correct periods and metrics. > "Check if metrics are being collected in time buckets... Each bucket should represent one time bucket (e.g., daily metrics)" [Anomaly Troubleshooting Guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide) 3. Test Configuration: Your test configuration appears correct for detecting changes in zero count. However, if you want to ensure that the absence of zeros is always flagged, you might consider using the
fail_on_zero
parameter, which is designed for this use case: > "If you're worried about the 0 values, you can configure the fail_on_zero test monitor." [Slack thread](https://slack.com/archives/C02CTC89LAX/p1736937571030059) In summary: Since your timestamp column is functioning for other tests, the issue is likely due to the statistical properties of your training data or the sensitivity threshold. Review the metrics in your
data_monitoring_metrics
and
metrics_anomaly_score
tables to see how the anomaly is being calculated. If you need stricter detection for zeros, consider using
fail_on_zero
as an additional safeguard.
o
Anyone who encountered this problem - it's due to the way you define the column anomalies. You're meant to define them in a list like this:
Copy code
- name: has_been_subscribed
  data_tests:
    - elementary.column_anomalies:
        column_anomalies:
          - zero_count
        timestamp_column: created_timestamp
        anomaly_sensitivity: 1
        time_bucket:
          period: day
          count: 1
        training_period:
          period: day
          count: 14
        detection_period:
          period: day
          count: 2