New message text here

@kapa.ai Should I refer to the timestamp field literally by

metric_date

or use the name of the field I specified with the

timestamp_column:

config?

New message text here

Yes, your understanding is correct. When using the

anomaly_exclude_metrics

parameter, the difference between

metric_date

and

metric_time_bucket

is their granularity: -

metric_date

is a date value (without time) representing the date of the relevant bucket -

metric_time_bucket

is a timestamp value (with date and time) representing the exact time bucket You should choose between them based on the level of precision you need for excluding metrics: 1. Use

metric_date

when you want to exclude entire days from your training data, regardless of the time bucket configuration. This is simpler when you're dealing with daily-level exclusions. 2. Use

metric_time_bucket

when you need more precise control, especially if you're using hourly or other sub-daily time buckets. This allows you to exclude specific time windows within days. For example:

yaml # Excluding entire days anomaly_exclude_metrics: metric_date >= '2023-10-01' and metric_date <= '2023-10-05'  # Excluding specific hours anomaly_exclude_metrics: metric_time_bucket >= '2023-10-01 06:00:00' and metric_time_bucket <= '2023-10-01 07:00:00'

The documentation shows examples of both approaches, with

metric_time_bucket

being used for more precise time-based filtering as seen in the example with hourly exclusions. [Elementary Documentation: anomaly_exclude_metrics](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-exclude-metrics)

Yes, you can combine both

metric_date

and

metric_time_bucket

in the same

anomaly_exclude_metrics

SQL statement using standard SQL operators like

OR

to handle multiple separate incidents. Here's how you can exclude entire days and specific time periods from different days:

yaml - elementary.column_anomalies:     column_name: your_column_name     anomaly_exclude_metrics: (metric_date >= '2023-10-01' and metric_date <= '2023-10-03') OR                               (metric_time_bucket >= '2023-10-15 13:00:00' and metric_time_bucket <= '2023-10-15 16:00:00')

This example excludes: 1. Three full days (Oct 1-3, 2023) 2. A specific 3-hour window on Oct 15, 2023 (from 1 PM to 4 PM) You can add as many conditions as needed using SQL's logical operators to handle multiple incidents. The

anomaly_exclude_metrics

parameter accepts any valid SQL where expression that can reference the fields

metric_date

,

metric_time_bucket

, and

metric_value

. [Elementary Documentation: anomaly_exclude_metrics](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-exclude-metrics) This approach gives you flexibility to exclude various time periods from your training data while keeping your anomaly detection thresholds accurate and preventing past incidents from influencing future anomaly detection.

I can see a few potential issues with your configuration that might be preventing the

anomaly_exclude_metrics

from working as expected: 1. Test location: In your YAML, you've placed the test under

data_tests

instead of

tests

. Elementary tests should be under the

tests

key:

yaml columns:   - name: field_name     tests:  # Should be "tests" not "data_tests"       - elementary.column_anomalies:

2. Anomaly sensitivity configuration: The issue might be related to how the anomaly detection algorithm works. According to the documentation, the detection period is actually included in the training period by default, which could be affecting your expected range:

i think i found the responsible lines in the code as well, shouldn't it be between unbounded preceeding and 1 preceding in these windows?

This is a known issue in Elementary, as mentioned in [this GitHub issue](https://github.com/elementary-data/elementary/issues/1491), where the detection period is included in the training period calculation. 3. Date format: Make sure your date format in the

anomaly_exclude_metrics

matches exactly what's in the metrics table. If your warehouse uses a different date format, this could cause issues. 4. Verification: To verify if your exclusion is working, you can check the Elementary metrics table directly to see if the metrics for those dates are being excluded from the anomaly score calculation. Here's a corrected version of your configuration:

yaml columns:   - name: field_name     tests:       - elementary.column_anomalies:           name: table_name__field_name           anomaly_sensitivity: 3           anomaly_direction: both           column_anomalies:             - null_percent           timestamp_column: event_date           time_bucket:             period: day             count: 1           training_period:             period: day             count: 21           detection_period:             period: day             count: 1           seasonality: day_of_week           anomaly_exclude_metrics: "metric_date >= '2025-03-24' and metric_date <= '2025-04-06'"

If this still doesn't work, you might need to try a more complex workaround as suggested in the GitHub issue:

yaml anomaly_exclude_metrics: "MAX(bucket_end) OVER (partition by metric_name, full_table_name, column_name, dimension, dimension_value, bucket_seasonality order by bucket_end asc rows between unbounded preceding and unbounded following) = bucket_end"

This is a hack to exclude the latest data point from the training period, which might help in your case.

To check the date format in the metrics table for your

anomaly_exclude_metrics

configuration, you should look at the

data_monitoring_metrics

table in your Elementary schema. This is the table where Elementary stores all the metrics data that's used for anomaly detection. You can run a simple query to check the format of dates in this table:

sql SELECT      bucket_end,     metric_value FROM your_schema.data_monitoring_metrics WHERE full_table_name = 'YOUR_TABLE_NAME'     AND column_name = 'field_name'     AND metric_name = 'null_percent' ORDER BY bucket_end DESC LIMIT 10;

Replace

your_schema

with your Elementary schema name (typically

elementary

) and

YOUR_TABLE_NAME

with the fully qualified name of your table. This query will show you the exact format of the

bucket_end

timestamps in the metrics table, which corresponds to the

metric_time_bucket

you can use in your

anomaly_exclude_metrics

expression. You can also see the date portion to understand how

metric_date

would be formatted. According to the documentation and Slack forum discussions, the metrics data is stored in this table and then used for anomaly calculations: [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1666376716952649) mentions: "The metrics are aggregated to

data_monitoring_metrics

and test results (anomaly or no) are in `elementary_test_results`" Make sure the date format in your

anomaly_exclude_metrics

expression exactly matches what you see in this table to ensure your exclusion works correctly.

I understand your concern about not being able to visualize the excluded time periods in the UI. This is indeed a limitation of the current

anomaly_exclude_metrics

parameter. Based on the knowledge sources, this is a known limitation that several users have encountered. When you use

anomaly_exclude_metrics

, the excluded metrics are filtered out from both the training period calculations AND from being displayed in the UI. There are a few GitHub issues that discuss this exact problem: 1. In [issue #1589](https://github.com/elementary-data/elementary/issues/1589), a user mentions: "By correctly setting

anomaly_exclude_metrics

, is it possible to achieve the same but the point will not be visualized in the dashboard." 2. Another user in [issue #980](https://github.com/elementary-data/elementary/issues/980) (now closed) requested a feature to "Allow a test to specify a training data filter to prevent over fitting thresholds to previous incidents" which was addressed with the

anomaly_exclude_metrics

parameter, but with the limitation you're experiencing. Currently, there doesn't appear to be a way to both exclude metrics from the training period while keeping them visible in the UI. This seems to be a feature gap in the current implementation. As for the product roadmap, the knowledge sources don't explicitly mention plans to address this specific limitation. However, there are several open issues related to improving anomaly detection and training data handling, which suggests the Elementary team is aware of these limitations and may be working on improvements. If this feature is important to you, I'd recommend: 1. Checking the [Elementary GitHub issues](https://github.com/elementary-data/elementary/issues) for updates 2. Potentially opening a new feature request that specifically addresses the need to exclude metrics from training while keeping them visible in the UI 3. Engaging with the Elementary team on their Slack community to ask about roadmap plans This would be a valuable enhancement to the platform that would allow for better visualization of incidents while maintaining accurate anomaly detection.