Hey elementary Team 👋 Getting

freshness_anomalies - Freshness

warning and not sure why. The latest entry column

PUBLISHED_AT = 2024-01-27 02:29:46.755000000

in

MY_SCHEMA.MY_TABLE

. The Elementary tests were executed at

2024-01-27T02:43:49Z

using the command

test --select source:MY_SCHEMA

This is the

schema.yml

file:

sources:
  - name: MY_SCHEMA
    schema: MY_SCHEMA
    tables:
      - name: MY_TABLE
        meta:
          elementary:
            timestamp_column: "PUBLISHED_AT"
        tests:
          - elementary.freshness_anomalies:
              config:
                severity: warn
              detection_delay:
                period: hour
                count: 10

The warning is sent 2024-01-27 044954:

Test: freshness_anomalies - Freshness
Status: warn
Time: 2024-01-27 02:45:12
last update was at 2024-01-25 09:54:25.503, 0.06 hours ago. usually the table is updated within 0.16 hours.

When I look into the data, I see evenly distributed data every few seconds. Any idea what can cause this?

Hey @Ilya Michlin !! Can you share the characteristics of this table, how frequent it's getting updated ? Notice that you don't mention a time bucket for this test, therefore we're using the default which is daily buckets

Hey @Daniel Pollak 🙂 This table is updated daily and adds events from the latest

PUBLISHED_AT

to the latest event. So practically it computes one day every day. There is an Airflow DAG in which the 1st task computes and inserts the new data. The 2nd task runs the tests using the command mentioned above. I do need the daily time buckets so that is why I did not changed it.

So, there suppose to be one record every day ? Can you share also please the last

PUBLISHED_AT

values from the last two weeks from this table ?

There are thousands of records every day. There is a record every few seconds. Let me clarify the data pipeline. The 1st task of the DAG, computes thousands of records every day from multiple sources. The values of

PUBLISHED_AT

are evenly distrebuted.

Ok, got it. As I understand the warning, it indicates that data has been refreshed faster than usual (based on training set) With freshness, early refresh are not interesting as delayed refreshes, so what you can do is add this parameter for detecting only delayed data:

anomaly_direction: spike

Cool. I will try that. Thanks!

You welcome, lmk whether it worked 🙂

BTW, looking on the freshness_anomalies documentation,

anomaly_direction

is not part of configurations (it is just stated as a default).

yup, i'm still looking into it. notice that i made a mistake, you need

anomaly_direction: spike

and not drop

Updated that. Thanks

@Ilya Michlin I now think it's not the issue. I'd like to go through the metrics our package collects in order to understand deeply the freshness of your table. May I ask you to share with me the results of this query -

select * from
<elementary_schema>.data_monitoring_metrics
where lower(full_table_name) like '%<table_name>%'
and updated_at > '2024-01-01'

Replace

<elementary_schema>

with our schema and

<table_name>

with the table name we're trying to understand. The result doesn't contain any contents of the table besides information about its updates

data_monitoring_metrics.txt

Good morning So I'm suspecting that you had a real anomaly (in terms of our test logic) in 24/1, and it looks like we're reporting the last point (whether it's anomalous or not) in the detection period in that case, which is what you see in the message. Didn't get to make sure in 100% yet though, so I still have to look, but feels like the problem

Hey @Daniel Pollak, I added

anomaly_direction: spike

and did not get the warnings since then. I think this is what was missing. Thanks for the help! 👑