Hi, how can I exclude the detection period from th...
# community-support
a
Hi, how can I exclude the detection period from the training period? Details in thread
👀 1
I notice that the row_count is correct for
current_date() - 1
however, the average for the training period is 7 weeks prior inclusive of
current_date() - 1
. I was looking at
anomaly_exclude_metrics
but when I configure this it filters out the date for both the detection and training period.
Copy code
- elementary.volume_anomalies: # Monitors the row count of table over time per time bucket
              # Known issue that detection date is included in training period: <https://github.com/elementary-data/elementary/issues/1491>
              # The following suggestion excluded the detection date from both the detection and training period <https://github.com/elementary-data/elementary/issues/1491#issuecomment-2384317102>
              # anomaly_exclude_metrics: "MAX(bucket_end) OVER (partition by metric_name, full_table_name, column_name, dimension, dimension_value, bucket_seasonality order by bucket_end asc rows between unbounded preceding and unbounded following) = bucket_end"
              description: |
                An anomaly detection test to compare volume of tickets created during the last complete day, against the
                average of the last 7 weeks for that same day (within 3 standard deviations). This is because volumes 
                vastly differ depending on the day of the week.
              severity: warn
              timestamp_column: created_at # We could use received_at, but we want to capture volume of new tickets
              anomaly_sensitivity: 3 # Default
              seasonality: day_of_week # Volumes differ day to day so we want to compare against the same day of the week
              detection_period: # We want to detect anomalies in volume from the last complete day
                period: day
                count: 1
              training_period: # We want a larger training period to detect anomalies against the previous day
                period: week
                count: 1 # When seasonality: day_of_week is configured, training_period is by default multiplied by 7.
              time_bucket: # We want to count new rows per day
                period: day
                count: 1
              tags: [ 'elementary' ]
Perhaps more readable
Elementary result:
Copy code
The last row_count value is 46409.000. The average for this metric is 44981.000.
Count of yesterday's rows: 46409
Copy code
select count(1) from my_table 
where created_at::date = '2025-02-19'; 
-- 46409
Average over last 7 weeks including yesterday: 44981
Copy code
with cte as (
    select 
        created_at::date as created_dt,
        count(1) as c 
    from my_table 
    where 1=1 
    and created_dt > '2025-02-19'::date - 7*7
    and dayname(created_dt) = 'Wed'
    group by created_dt
    order by created_dt
)
select avg(c)::int from cte
;
Average over last 7 weeks excluding yesterday: 44743 (6 days of data)
Copy code
with cte as (
    select 
        created_at::date as created_dt,
        count(1) as c 
    from my_table 
    where 1=1 
    and created_dt < '2025-02-19'
    and created_dt > '2025-02-19'::date - 7*7
    and dayname(created_dt) = 'Wed'
    group by created_dt
    order by created_dt
)
select avg(c)::int from cte
;
I would expect the training set to not include the detection date
Hi is somebody able to please help me with this? Thanks
🙌 1
n
Hey Amy, thank you for bringing this up! I understand why you would want to separate the training and detection periods, but there's no workaround for this with the existing Elementary parameters. We will look into this some more to find a solution 🙂
a
Hi Noya, thanks for your reply! Is there anything I can follow to track the progress of this feature? 🙂
n
We'll take this into our planning an update here (in #C02BPB78J4T) when it's done 😊
a
Amazing thank you!
drop stoked 1
🙌 1
Hey @Noya Offer I just wondered if there was any update on this? Thanks 🙂
n
Hey @Amy G! Sorry to keep you waiting for an answer, this is still in discussions. We're planning to work on improving anomaly detection over the next few weeks, so this should be a part of that effort. I'll let you know when we have a better timeline!
a
No worries thanks for the update!
j
Hi @Amy G! Just letting you know we are introducing a feature flag for this. In the next release this will be available. Curious in your feedback once we release it
a
Amazing thanks so much for following up on this!