Hi <@U088WCP3B4P> I’m struggling to reduce the que...
# ask-elementary-ai
a
Hi @kapa.ai I’m struggling to reduce the query cost for a dimension_anomalies test. I am time-bucketing hourly and want to have the seasonality as hour_of_day, but that seems to lookback weeks which is very expensive on my enormous table. Can you help me understand why the test is recalculating data from days ago when I only want to detect today?
k
A message from kapa.ai
When you configure a dimension_anomalies test with hourly time buckets and set seasonality to hour_of_day, Elementary will compare each hour to the same hour in previous days (e.g., 10AM today to 10AM on previous days). To do this, it needs enough historical data for each hour to build a reliable training set and avoid false positives. By default, Elementary multiplies the training period by 7 when using day_of_week seasonality, and a similar logic applies for hour_of_day: it extends the lookback window to ensure there are enough data points for each hour slot. This means that even if you set a short detection period, the query will scan a much larger historical window to gather enough training data for each hour, which can be expensive on large tables. This is a product decision to ensure statistical reliability and avoid spamming you with false positives when there isn’t enough data for each time bucket and seasonality combination [seasonality documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality), [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1689000364837139). If you want to reduce query cost, you can try: - Lowering the days_back or training_period value in your test config, but keep in mind that Elementary will still multiply this by 7 (or more) for seasonality, so the effective lookback is much longer than the raw value you set [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1696533226544519). - Pre-aggregating your data into a smaller table with the relevant time buckets before running the anomaly test, as some users have done to improve performance [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1703054856878699). Unfortunately, there is currently no way to force Elementary to only look at today’s data for anomaly detection with seasonality, as the training set must include enough historical data for each hour to be statistically meaningful. This is a known limitation and has been discussed in the community [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1689000364837139). If you need more granular control or want to avoid scanning large tables, consider pre-aggregating or filtering your data as much as possible before running the test.
a
@kapa.ai I’m not sure I follow, why does elementary store the data_monitoring_metrics if it is just going to be recalculating the training data every time?
k
A message from kapa.ai
Elementary stores metrics in the
data_monitoring_metrics
table specifically to avoid recalculating historical training data every time a test runs. When an anomaly detection test executes, it queries this table for historical metrics and only needs to calculate new metrics for the latest data since the last run. This incremental approach is designed to reduce query costs and improve efficiency, especially on large tables. The historical metrics are used as the training set for anomaly detection, so only new or missing data points are computed, not the entire history each time [Elementary dbt package models](https://docs.elementary-data.com/dbt/package-models#anomaly-detection). If you notice recalculation of older data, it may be due to configuration issues (such as changes in the timestamp column or a full refresh), but the intended behavior is to leverage the stored metrics to minimize redundant computation [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1650989426047689).
a
@kapa.ai I think part of my problem is that the metric_value is showing row counts in my dimension_anomaly test when I was expecting a percentage of total
k
A message from kapa.ai
You're correct: in Elementary's current implementation, the
dimension_anomalies
test calculates and monitors the row count per dimension value, not the percentage of total or distribution share. The metric stored and analyzed is the absolute count of rows for each unique value in the specified dimension(s) over time. This is explicitly described in the documentation and confirmed in multiple community discussions: > The test counts rows grouped by given
dimensions
(columns/expressions). > This test practically monitors the frequency of values in the configured dimension over time, and alerts on unexpected changes in the distribution. > It is best to configure it on low-cardinality fields. > If
timestamp_column
is configured, the distribution is collected per
time_bucket
. If not, it counts the total rows per dimension. [dimension_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies) Several users have requested the ability to monitor percentages or more complex metrics (like share of total, or custom aggregations) per dimension, but as of now, this is not supported. The test only tracks row counts, not proportions or percentages of the total. This limitation is also discussed in community threads and GitHub issues, where the need for percentage-based or custom metric monitoring by dimension is a known feature request, but not yet available [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671712508592539), [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1678792941742089). If you need to monitor the percentage of total per dimension, you would currently need to pre-aggregate your data or create a custom model that calculates these percentages, and then run anomaly detection on that model using a different test (such as
column_anomalies
), but this is a workaround and not natively supported by
dimension_anomalies
. Let me know if you need guidance on how to set up such a workaround!
a
That’s super unclear in the documentation as “unexpected changes in the distribution” sounds like a distribution of the values occurring rather than as row count.