To perform anomaly detection on an incremental table that runs every hour using the

activity_date

as the timestamp column, you can configure the

elementary.volume_anomalies

test. This test will help you detect anomalies in the row count based on the past 7 hours. Here's how you can set it up:

yaml models:   - name: <your_model_name>     tests:       - elementary.volume_anomalies:           timestamp_column: activity_date           time_bucket:             period: hour             count: 1           training_period:             period: hour             count: 7

### Explanation: -

timestamp_column: activity_date

: This specifies the column used to track time in your data. -

time_bucket

: Configured to

hour

with a count of

1

, meaning the data is bucketed by each hour. -

training_period

: Set to the last 7 hours, which means the test will compare the current hour's row count against the average of the past 7 hours to detect anomalies. This configuration will help you monitor the row count on an hourly basis and alert you if the count deviates significantly from the expected range based on the past 7 hours. For more details, you can refer to the [Volume Anomalies Test Configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies#test-configuration).

@kapa.ai if I run the dbt test now, will elementary calculates the past 7 hours*`training_period ?`* hence it will detect if the latest dbt test is anomalous?

@kapa.ai if I run the dbt test now, will elementary calculates the past 7 hours*`training_period ?`* hence it will detect if the latest dbt test is anomalous?

When you run the dbt test with the configuration provided, Elementary will calculate the row counts for the past 7 hours as specified by the

training_period

. This means that it will use the data from these past 7 hours to establish a baseline for what is considered "normal" activity. If the row count from the latest run deviates significantly from this baseline, it will be flagged as an anomaly. The test will only consider completed time buckets, so it will compare the current hour's data against the established baseline from the previous 7 hours. For more information on how the training period works, you can refer to the [Training Period Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period#how-it-works%3F).

To perform anomaly detection based on the sum of the

accounting_revenue_eur

column in an incremental table that runs every hour, you can configure the

elementary.column_anomalies

test. This test will help you detect anomalies in the sum of the specified column by comparing it with the past 7 hours. Here's how you can set it up:

yaml models:   - name: <your_model_name>     columns:       - name: accounting_revenue_eur         tests:           - elementary.column_anomalies:               column_anomalies:                 - sum               timestamp_column: activity_date               time_bucket:                 period: hour                 count: 1               training_period:                 period: hour                 count: 7

### Explanation: -

column_anomalies: sum

: This specifies that the test should monitor the sum of the

accounting_revenue_eur

column. -

timestamp_column: activity_date

: This specifies the column used to track time in your data. -

time_bucket

: Configured to

hour

with a count of

1

, meaning the data is bucketed by each hour. -

training_period

: Set to the last 7 hours, which means the test will compare the current hour's sum against the average of the past 7 hours to detect anomalies. This configuration will help you monitor the sum of

accounting_revenue_eur

on an hourly basis and alert you if the sum deviates significantly from the expected range based on the past 7 hours. For more details, you can refer to the [Column Anomalies Test Configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies#test-configuration).

In the context of an incremental model, the

training_period

parameter in Elementary's anomaly detection tests is used to define the historical data range that establishes the baseline for what is considered "normal" behavior. Here's how it works: ### How

training_period

Works: - Incremental Models: For incremental models, the values of the full

training_period

are calculated on the first test run and during a full refresh. Subsequent test runs will only calculate the values for the

detection_period

. - Baseline Calculation: The baseline is established using the data from the specified

training_period

. This baseline is used to compare against the current data to detect anomalies. ### Handling Anomalies: - Inclusion of Anomalies: If an anomaly is detected in one of the past runs, it will be included in the calculation of the baseline moving forward. This means that the anomaly will affect the average and standard deviation calculations used to determine what is considered "normal" in future runs. - Impact on Future Anomalies: Including anomalies in the baseline can potentially skew the baseline, making it less sensitive to future anomalies. This is a known behavior, and there are discussions about potentially changing this approach to exclude anomalies from the baseline calculation. For more detailed information, you can refer to the [Training Period Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period#how-it-works%3F).

To exclude anomalies from the training dataset, one approach is to adjust the data that caused the anomaly. However, there are other methods you can consider: ### Workarounds to Exclude Anomalies: 1. Fix the Data: Correct the data that caused the anomaly if it was due to an error. This ensures that the baseline is calculated using accurate data. 2. Use

anomaly_exclude_metrics

: You can configure the test to exclude specific data points from the training set using the

anomaly_exclude_metrics

parameter. This allows you to specify conditions under which certain data points should be ignored in the baseline calculation.

yaml    models:      - name: <your_model_name>        tests:          - elementary.volume_anomalies:              anomaly_exclude_metrics: metric_value < 10

This example excludes metrics where the

metric_value

is less than 10 from the training set. 3. Provide Feedback: If using the cloud version, you can provide feedback on false positives, which can help the system learn and adjust the model accordingly. This feature is not available in the open-source version. For more details on excluding metrics, you can refer to the [Anomaly Exclude Metrics Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-exclude-metrics). These methods can help maintain the accuracy of your anomaly detection by ensuring that the baseline reflects normal, expected data patterns.