<@U088WCP3B4P> How to create a volume anomaly test...
# ask-elementary-ai
r
@kapa.ai How to create a volume anomaly test that checks, right from the next run, if the table has fewer records than the previous count? I would like to only accept if the table has greater than or equal row count than the previous execution
k
A message from kapa.ai
To create a volume anomaly test in Elementary that checks if the table has fewer records than the previous count (i.e., only accept if the row count is greater than or equal to the previous execution), you should use the elementary.volume_anomalies test without specifying a timestamp_column. This configuration will make the test compare the total row count of the table at each run, rather than splitting by time buckets. Here’s how it works: - When you omit the timestamp_column, Elementary saves the entire row count of the table at each test run. - After enough runs (the default training period is 14 runs, but you can configure this to be shorter), the test will start comparing the current row count to the previous runs. - If the row count drops compared to the previous executions, the test will flag it as an anomaly. Example configuration:
Copy code
yaml models:   - name: your_model_name     tests:       - elementary.volume_anomalies
Key points: - Do not set timestamp_column. - The test will take a "snapshot" of the row count at each run and compare it to previous snapshots. - You may want to reduce the training period (e.g., days_back) if you want the test to start flagging anomalies sooner than the default 14 runs. Reference: This approach is described in detail by Elementary support: > "If you don't define a timestamp column for this table, we will simply save the entire row count of the table when the test runs... once we have enough data we can use it to alert on anomaly" [Slack thread](https://slack.com/archives/C02CTC89LAX/p1650989426047689). Also, the documentation confirms that omitting timestamp_column will count the table's total rows and compare them across runs [volume_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). If you want the test to be sensitive to any drop (not just statistical anomalies), you may need to adjust the anomaly_sensitivity parameter, but the default behavior should already flag significant drops. Note: If you need to flag even a single-row drop (i.e., strictly greater than or equal), Elementary’s anomaly detection is based on statistical analysis and may not catch very small changes unless configured for high sensitivity. For strict constraints (e.g., "row count must never decrease"), you might also consider using a dbt-expectations test like expect_table_row_count_to_be_between, which can be included in Elementary reports as well [Slack thread](https://slack.com/archives/C02CTC89LAX/p1677563837770559).