Is there a setting that will allow me to detect anomalies same day? When running the volume anomalies test today (23/1/24) the bucket shown on the report hovering over today is (21/1/24 000000 - 22/1/24 000000). This shows me yesterday's anomaly. Do I need to wait until tomorrow to see the volume results for today's data load?

- elementary.volume_anomalies:

timestamp_column: "process_ended_at_utc"

fail_on_zero: true

Hi @Damian Eralio, at the time of running the test, do you have data with

process_endedn_at_utc > '2024-01-23'

?

@Ella Katz When i was working on this yesterday at around 8 - 10 PM UTC, there were records loaded a few hours prior, at different times of the day. Their

process_ended_at_utc

would therefore be

= '2024-01-23'

not

> '2024-01-23'

Of course, if we use timestamp instead of date format, those records are

> '2024-01-23 00:00:00'

So I think this is because we're only checking the test for buckets that have ended already. What I think you can do as a workaround is define your time bucket as 24 hours instead of 1 day, this way it'll go backwards 24 hours and split the buckets in that way. WDYT?

I can understand the concept of testing buckets only when they are complete. I thought about your suggestion, and what I like about Elementary is that if 1h increments didn’t work for us, we could probably control that by using something like 8h buckets. This is probably a very restrictive constraint for my purposes. While it gives results quicker, it will also likely lead to more alert fatigue. Thanks for confirming and talking this out with me.thankyou I still think the team should consider how confusing it looks to read 0 on

2024-01-23

only to look closer and find out that the data tested on that date is for the day prior. Something to consider.

sure thing! Keep me posted 🙂

Just wanted to reach back out with some results. Using a 24 hour period instead rendered the exact same buckets as 1 day: Using 8 hour periods instead broke the buckets into even 8 hour periods:

00:00:00 - 08:00:00

08:00:00 - 16:00:00

16:00:00 - 00:00:00

This test was run at 113738 MT, which is 163738 UTC. So 2 or the 3 buckets for today have been completed and show up on the results. Obviously this small time bucket will lead to more alert fatigue. If the goal was to catch runs with 0 results, we are now generating a lot of false positives as we expect a number of those 8 hour buckets to have 0 results. Perhaps we should be leaning on the dbt freshness test or looking at creating our own generic dbt test ("no new records have come in so far today")

Hi @Damian Eralio Thanks for providing this info! As for the 24 hour buckets, I now understand that this is expected behavior, we want to split the buckets evenly across the day and not depend on the hour that the test ran because we don't want to re-calculate all the training period each time. As for how to work around this, If I understand you correctly, there's a certain hour by which you would expect to see some rows in the table and if there are none you want to be alerted as soon as possible. here are 3 possible solutions I can suggest for you- 1. use daily buckets and run the test at 00:01 every day- this is the earliest you can be alerted on the previous day 2. use hourly buckets but add seasonality of hour-of-day. assuming the data loads on the same time each day, most of your hours will be steady on 0 so it won't trigger an alert. The downside is this will not be tolerant to delays in data of more than 1 hour, but it will be able to alert you as soon as possible 3. You can leave the volume anomaly test to check if there's a spike or drop in the row count, but to get an alert on data delays sooner you can add an additional recency test that isn't an anomaly-detection test- there are recency tests on both dbt_utils and dbt_expectations (

expect_row_values_to_have_recent_data

for example)

Thanks for your great suggestions. I have implemented the default dbt freshness tests to keep track of alerts regarding data not coming in as expected. Although those tests don't appear to be supported in the Elementary dashboard, we are getting alerts through elementary on them.

Hi @Damian Eralio, you are right that freshness tests were not supported in the elementary dashboard, but support for them was added in a recent update (drop stoked!!), what version of

edr

are you using?