Hello <@U088WCP3B4P> ! We have a few anomaly detec...
# ask-elementary-ai
d
Hello @kapa.ai ! We have a few anomaly detection tests on our project and our monitoring_data_metrics has become huge because it keeps historical data from years ago. We’ve set the training_period for our tests, and we know that by full-refreshing the table we can reset it, however we were wondering if there is an option to prevent the table from growing exponentially?
k
A message from kapa.ai
Short answer: there’s no documented built‑in retention/auto‑cleanup for data_monitoring_metrics. You can control how much is recalculated, but not how long historical rows are kept unless you reset or delete them. Practical options: - Periodic full refresh of metrics: dbt run --select data_monitoring_metrics --full-refresh, then rerun tests. This rebuilds the metrics table for your configured training_period/detection_period. [[Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] - Manual pruning: delete old rows from data_monitoring_metrics (e.g., by date or table) and keep only recent history you need. Elementary stores all metrics there and supports deleting rows you no longer want. [[Slack cleanup tip](https://slack.com/archives/C02CTC89LAX/p1681289966274379)] - Limit recalculation scope: training_period and detection_period restrict what’s collected/recomputed, but they do not automatically purge older rows already stored (especially for incremental models, where only detection_period is recalculated after the first run). [[training_period](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period); [detection_period](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period)] Note: The edr report --days-back flag only limits what’s shown in the HTML/JSON report, not what’s stored in your warehouse. [[General troubleshooting](https://docs.elementary-data.com/oss/general/troubleshooting)]
d
@kapa.ai thank you! From what you know about the codebase, do you think it could make sense to add this as an option?
k
A message from kapa.ai
Yes—it’s a reasonable addition. Today there’s no built‑in retention/lifecycle policy, and teams either delete old rows themselves or wire cleanup into an on‑run‑end hook. [[No lifecycle policy](https://slack.com/archives/C02CTC89LAX/p1689146074949549); [Manual cleanup/on-run-end](https://slack.com/archives/C02CTC89LAX/p1701698313474989)] From a package-design standpoint, a retention option (e.g., keep last N days) is feasible: pruning rows in warehouse tables by a date column is low‑risk and consistent with current guidance to manually delete old records. There’s no CLI flag for general cleanup today (only an alerts-specific flag), which also suggests room for a configurable retention feature. [[CLI cleanup scope](https://slack.com/archives/C02CTC89LAX/p1719390641107289); [Delete old rows guidance](https://slack.com/archives/C02CTC89LAX/p1683451667062879)] If you’d like this prioritized, please open or upvote a feature request—cleanup-focused requests have been accepted before (e.g., auto-deleting temporary test tables). [[Temp tables cleanup](https://github.com/elementary-data/elementary/issues/937)]