@kapa.ai I have a table that has volume anomaly tests defined (alert on drop), and we have some expected changes to the source data that will reduce the volume in an expected way. This will cause tests to fail if I don't do something, is there a recommended approach for dealing with expected data loss for volume anomaly tests that allows us to still run the pipeline?

There are a few recommended approaches for handling expected volume drops in Elementary's

volume_anomalies

test: 1. Use anomaly_exclude_metrics (Best for known time windows) If you know when the expected data reduction will occur, you can exclude those specific data points from the training set using a SQL

WHERE

expression. For example:

models:
  - name: your_model_name
    tests:
      - elementary.volume_anomalies:
          anomaly_direction: drop
          anomaly_exclude_metrics: "metric_date = '2024-06-01'"

Or exclude by a value threshold (e.g., if the reduced volume drops below a known floor):

anomaly_exclude_metrics: "metric_value < 500"

This prevents the anomalous data points from skewing the training set for future runs. [anomaly_exclude_metrics discussion] 2. Use ignore_small_changes (Best for stable tables with predictable % changes) If the expected volume reduction is within a known percentage range, you can configure the test to only fail if the drop exceeds that threshold:

models:
  - name: your_model_name
    tests:
      - elementary.volume_anomalies:
          anomaly_direction: drop
          ignore_small_changes:
            drop_failure_percent_threshold: 50  # Only fail if drop exceeds 50% of mean

This is most appropriate when your data is relatively stable and you want to avoid false positives from changes within an acceptable range. [ignore_small_changes docs] Key Consideration Note that by default, failed data points are included in future training runs. So if the volume drop is a one-time event, the anomaly detection will eventually self-correct as the new lower volume becomes part of the training baseline. However, if you want to prevent the reduced-volume period from permanently skewing the model,

anomaly_exclude_metrics

is the more precise tool. [failed data points in training] The Elementary Cloud product can learn these patterns automatically, but that capability is not available in the open-source version. [OSS limitation] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.