https://linen.dev logo
Join Slack
Powered by
Hey all! I'd love some support, I'm migrating fro...
# ask-elementary-ai
n
Hey all! I'd love some support, I'm migrating from another testing package to Elementary so I just want to re-check my config as it doesn't seem to be behaving. I have a 
agg_metrics
table that has a date dimension 
event_date_dt
, I want to set up anomaly detection on all columns in this table. This is my config:
Copy code
models:
  - name: agg_metrics
    config:
        elementary:
            timestamp_column: event_date_dt
            time_bucket:
                period: day # supported periods: day, week, month
                count: 1
            anomaly_sensitivity: 2.2
            anomaly_direction: both
            training_period:
                period: day # supported periods: day, week, month
                count: 90
            detection_period:
                period: day # supported periods: day, week, month
                count: 30
            seasonality: day_of_week
    tests:
      - elementary.all_columns_anomalies: 
            column_anomalies:
                - standard_deviation
                - variance
                - zero_count
                - zero_percent
This is the command I'm running locally (windows):
Copy code
dbt run -m agg_metrics && dbt test -m agg_metrics && edr report --profiles-dir . --profile-target dev
My report doesn't seem to calculate standard_deviation or variance, and the training period is 338 days. Any ideas on what I'm doing wrong?
@kapa.ai please help
k
A message from kapa.ai
👍 1
New message text here
n
I'd love some support, I'm migrating from another testing package to Elementary so I just want to re-check my config as it doesn't seem to be behaving. I have a 
agg_metrics
table that has a date dimension 
event_date_dt
, I want to set up anomaly detection on all columns in this table. This is my config:
Copy code
models:
  - name: agg_metrics
    config:
        elementary:
            timestamp_column: event_date_dt
            time_bucket:
                period: day # supported periods: day, week, month
                count: 1
            anomaly_sensitivity: 2.2
            anomaly_direction: both
            training_period:
                period: day # supported periods: day, week, month
                count: 90
            detection_period:
                period: day # supported periods: day, week, month
                count: 30
            seasonality: day_of_week
    tests:
      - elementary.all_columns_anomalies: 
            column_anomalies:
                - standard_deviation
                - variance
                - zero_count
                - zero_percent
This is the command I'm running locally (windows):
Copy code
dbt run -m agg_metrics && dbt test -m agg_metrics && edr report --profiles-dir . --profile-target dev
My report doesn't seem to calculate standard_deviation or variance, and the training period is 338 days. Any ideas on what I'm doing wrong?
@kapa.ai how does detection_period work?
k
A message from kapa.ai
The 
detection_period
in Elementary Data is a key configuration parameter for anomaly detection tests that determines which data points will be evaluated for anomalies. ## How detection_period works When you run an anomaly detection test with a configured 
timestamp_column
, the 
detection_period
defines the time window for which Elementary will look for anomalies. Only data points within this period will be flagged as anomalous if they fall outside the expected range calculated from the training data. For example, if your 
detection_period
is set to 2 days (the default), Elementary will only check for anomalies in data from the last 2 days. If set to 7 days, it will examine data from the last week. The behavior differs slightly based on table materialization: - For regular tables and views: 
detection_period
simply defines the detection window - For incremental models and sources: 
detection_period
defines both the detection window and the period for which metrics will be recalculated [How it works?](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period#how-it-works%3F) ## Configuration example
Copy code
yaml models:   - name: this_is_a_model     tests:       - elementary.volume_anomalies:           detection_period:             period: day             count: 30
This configuration would set the detection period to 30 days, meaning Elementary would look for volume anomalies in the last 30 days of data. It's important to note that 
detection_period
works in conjunction with 
training_period
. While the 
training_period
(default: 14 days) determines what data is used to establish "normal" behavior, the 
detection_period
determines what data is checked against that normal baseline to identify anomalies. As one Elementary staff member explained: "Both are required for the anomaly test to function: training_period is how long back to look in order to understand what is the 'normal' behavior of the table, and detection_period is how long back to look for data that is not 'normal' according to the behavior that was detected during the training_period." [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1709876086206929)
n
Does this mean @kapa.ai that if the 
detection_period
is 30 days, and I had an anomaly yesterday, I'll have alerts for the next 29 days?
k
A message from kapa.ai
No, setting a 
detection_period
of 30 days doesn't mean you'll receive alerts for the next 29 days after detecting an anomaly. The 
detection_period
simply defines the window of time that Elementary Data will examine for anomalies when the test runs. To control how frequently you receive alerts for the same issue, Elementary Data provides a separate configuration called 
alert_suppression_interval
. This parameter defines how long Elementary will wait before sending another alert about the same test failure. As explained by an Elementary support staff member: > The 
alert_suppression_interval
defines an interval you won't get re-alerted for a test failure you've been already alerted for. That's our only option currently for controlling the alerts pace after a failure. [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1706451061267869) If you want to prevent repeated alerts for the same anomaly, you could set the 
alert_suppression_interval
to match your detection period. For example, if you set: - 
detection_period
to 2 days - 
alert_suppression_interval
to 48 (hours) This would effectively send a single alert per anomalous sample of the test, preventing repeated notifications for the same issue. The 
detection_period
is primarily about which data points are evaluated for anomalies, not about controlling alert frequency.
5 Views
Open in Slack
PreviousNext
Powered by