<@U088WCP3B4P> I'm trying to build a test that doe...
# ask-elementary-ai
j
@kapa.ai I'm trying to build a test that does the total row count of yesterday and flags whether the number looks unusual. Can you give me the skeleton code for the anomaly detection on this. I also want seasonality.
k
A message from kapa.ai
Based on the knowledge sources, here's the skeleton code for a volume anomaly detection test that monitors yesterday's row count with seasonality:
Copy code
yaml models:   - name: < model name >     tests:       - elementary.volume_anomalies:           timestamp_column: < timestamp column >           seasonality: day_of_week           time_bucket:             period: day             count: 1           detection_period:             period: day             count: 1           training_period:             period: week             count: 1
Here's what each parameter does: -
timestamp_column
: The column that represents when each row was created. This is highly recommended for time-based anomaly detection ([volume_anomalies test configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies#test-configuration)) -
seasonality: day_of_week
: Compares each day to previous instances of the same day of the week (e.g., Mondays to Mondays). This is useful when data patterns differ by day of week ([seasonality configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality)) -
time_bucket
: Set to daily buckets to count rows per day ([time_bucket configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket)) -
detection_period
: Set to 1 day to detect anomalies in yesterday's data ([how anomaly detection works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works)) -
training_period
: Set to 1 week, which when combined with
seasonality: day_of_week
, is automatically multiplied by 7 to provide sufficient training data ([seasonality configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality)) The test will only run on completed time buckets, so if you run it during the day, it will analyze yesterday's complete data ([volume_anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies)).
j
@kapa.ai Are you sure this will look at yesterdays row count?
k
A message from kapa.ai
Yes, I'm confident this configuration will look at yesterday's row count. Here's why: According to the [volume_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies), "The test will only run on completed time buckets, so if you run it with daily buckets in the middle of today, the test would only count yesterday as a complete bucket." This is further confirmed in a [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671002553673119) where Elementary staff explicitly states: "the logic is to only calculate anomalies for full days, meaning the latest day would be yesterday". With the configuration I provided: -
time_bucket: period: day, count: 1
creates daily buckets -
detection_period: period: day, count: 1
sets the detection window to the last complete day Since Elementary only evaluates completed time buckets, and today is incomplete until midnight, the test will analyze yesterday's data when you run it. However, there's an important caveat mentioned in a [GitHub issue](https://github.com/elementary-data/elementary/issues/1523): the current implementation may actually look at data from 2 days ago in some cases due to how bucket boundaries are calculated. Additionally, a [recent Slack discussion](https://slack.com/archives/C02CTC89LAX/p1740069258974599) notes that the detection date is currently included in the training period, which is a known limitation being considered for improvement. If you need to exclude even more recent data (e.g., if yesterday's data isn't fully loaded when your test runs), you can add a [
detection_delay
](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-delay) parameter to push back the detection window further.
j
@kapa.ai How many days do I have to run this before it will start alerting correctly? Does it take time to be trained?
k
A message from kapa.ai
Yes, the test does require a training period before it will start alerting correctly. Here's what you need to know: ## Training Period Requirements When using
seasonality: day_of_week
as in your configuration, you'll need to wait longer than the standard training period. According to the [seasonality documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality): "When
seasonality: day_of_week
is configured,
training_period
is by default multiplied by 7."
This is confirmed in a [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1701180350439159) where Elementary staff explains: "When configuring seasonality in a test, the days_back is per 'season' - so if day_of_week is configured with 30 days back you should get 30 days history per each day of the week, which is total 210 days." ## Specific Timeline Based on the [training period documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period): - Default training period: 14 days - With
seasonality: day_of_week
: This is automatically increased to ensure enough data points for each day of the week - Minimum data points needed: At least 7 training data points are required before anomaly calculations begin ([Anomaly Troubleshooting Guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)) ## How It Works As explained in a [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1690196920752279): "We're unable to detect anomalies in data if we don't have enough data accumulated and therefore we require by default 14 days of data with a minimum of 14 rows." During the training period, Elementary collects metrics but won't fail the test. Once sufficient data is accumulated, it will start comparing new data points against the historical baseline and alert on anomalies. ## Reducing the Training Period If you want to test sooner, you can reduce the
min_training_set_size
parameter, though this may affect accuracy ([Anomaly Troubleshooting Guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)).
j
@kapa.ai I have 1 years worth of data, surely it should work from straight away?
k
A message from kapa.ai
Yes, you're absolutely right! Since you have 1 year of historical data and you're using a
timestamp_column
, the test should work immediately from the first run. According to the [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column): "When you specify a
timestamp_column
, when the test runs it splits the data to buckets according to the timestamp in this column, calculates the metric for each bucket and checks for anomalies between these buckets. This also means that if the table has enough historical data, the test can start working right away."
This is further clarified in the [training period documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period): "Regular tables and views - The values of the full
training_period
period is calculated on each run."
## Why It Works Immediately With a
timestamp_column
configured: - Elementary looks back at your historical data based on the
training_period
setting - It creates time buckets from your existing data (in your case, daily buckets from the past year) - It immediately has enough training data to calculate anomalies - The test will work on the first run This is different from tests without a
timestamp_column
, which need to run multiple times over days/weeks to accumulate metrics, as explained in the [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column): "When you do not specify a
timestamp_column
, each time the test runs it calculates the metric for all of the data in the table, and checks for anomalies between the metric from previous runs. This also means that it will take the test
training_period
days to start working, as it needs to the time to collect the necessary metrics."
Since you have a year of data and are using a
timestamp_column
, your test should start detecting anomalies immediately on the first run!
j
@kapa.ai I would like seasonality please.
k
A message from kapa.ai
I believe there may be some confusion - the skeleton code I provided in my first response already includes seasonality! Here's the configuration again with the seasonality highlighted:
Copy code
yaml models:   - name: < model name >     tests:       - elementary.volume_anomalies:           timestamp_column: < timestamp column >           seasonality: day_of_week  # ← Seasonality is already configured here           time_bucket:             period: day             count: 1           detection_period:             period: day             count: 1           training_period:             period: week             count: 1
The
seasonality: day_of_week
parameter means that the test will compare each day to previous instances of the same day of the week. For example, Mondays will be compared to previous Mondays, Tuesdays to previous Tuesdays, and so on. According to the [seasonality documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality): "The
day_of_week
seasonality uses the same day of week as a training set for each daily time bucket data point. The expected range for Monday will be based on a training set of previous Mondays, and so on."
If you'd like a different type of seasonality, Elementary also supports: -
hour_of_day
- for hourly patterns within a day -
hour_of_week
- for hourly patterns across the week Is there a specific aspect of the seasonality configuration you'd like to adjust?