Working on trying to tune out some of the noise from our alerts (a problem that will always exist in some capacity 🙂 ) -- specifically, we have a some Dimension anomaly tests where we're looking at volume for an "account" dimension (for this purpose we have roughly a dozen accounts, so not an entirely unreasonable dimension for our use case). A couple of these accounts generate extremely low volume, think 0-2 records on a daily basis while others generate hundreds per day. So my problem specifically is that these small accounts might have an average of 0.5 records per day, and alert on a day when they get 2 records. Now I could simply use the where clause to filter these accounts out, but I actually would like to know if these accounts experience an unexpected spike, say maybe they hit 10 records. But I can't think of a way to execute this well today -- The

ignore_small_changes

feature is sort of the spirit of what I want, but while that solves for small % changes on extremely large/stable records, I need to solve for small # changes on smaller records. eg, I think ideally i would have a way to not alert unless the numerical delta between observed and average exceeded a certain range. So where we have

spike_failure_percent_threshold

, i almost need something like

spike_failure_absolute_threshold

where we require the delta between observed and average to exceed a certain value. Thought I'd share this to see if I was just maybe missing a way to configure this today or if it could be considered as a feature request 🙂

Hi @Christian Jones, Thank you for the detailed suggestion, we will definitely look into this! Would you mind opening a feature request in our Github for this issue? In the meantime, can I suggest you try to use a higher sensitivity for this test? It will obviously apply to all the accounts but still might be worth a try.

@ella katz will do! Yeah might try adjusting the sensitivity in the interim, just want to be careful that I don't over tune that and end up missing something we would have actually wanted to be alerted on. Always walking a fine line when it comes to alerts 😎

Yeah I hear you 😄 BTW- If there would be an absolute threshold, what would you choose that would fit all your accounts?

I'm not sure -- offhand I'm thinking maybe 20 or 25. I think anything with a delta smaller from the average I probably wouldn't want to know about. And for slightly larger accounts this gets into a territory where the standard z score approach is much less likely to produce false positives.

so, for accounts that get 0-2 records a day you would only get an alert if they hit 20-25 records? would that be ok for you?

@ella katz yeah that would be okay for me -- Again, not 100% sure if I would use 20-25, might realistically use something like 10, but that idea would be that whatever this number is, I don't really need/want to to be notified if a particular account were to hit anything less than that number. And good point on the other workaround you mentioned -- I think that actually I'd be more comfortable with than adjusting the sensitivity of the test. Really the only downside is that it's more cumbersome to manage as it require bifurcating the alerts configurations. But I have done something similar in other scenarios and it's worked pretty well for me.

Cool, let me know if the workaround fixed the issue for you 🙂 As for the absolute thresholds, we'll definitely consider adding that to our roadmap!