Hello I m wondering if what I m seeing with freshness alerts Elementary Community #community-support

Join Slack

Hello. I'm wondering if what I'm seeing with fresh...

# community-support

Francois Forster

12/13/2023, 8:06 PM

Hello. I'm wondering if what I'm seeing with freshness alerts on my data is a bug. Details in the thread

Francois Forster

12/13/2023, 8:08 PM

I have a job that write BQ data daily at the same time. Easy to reproduce with:

Copy code

create table `test_table` (timestamp timestamp, id int);

insert into `test_table`
(timestamp, id)
values
(timestamp("2023-11-27T12:00:00"), 123),
(timestamp("2023-11-28T12:00:00"), 123),
(timestamp("2023-11-29T12:00:00"), 123),
(timestamp("2023-11-30T12:00:00"), 123),
(timestamp("2023-12-01T12:00:00"), 123),
(timestamp("2023-12-02T12:00:00"), 123),
(timestamp("2023-12-03T12:00:00"), 123),
(timestamp("2023-12-04T12:00:00"), 123),
(timestamp("2023-12-05T12:00:00"), 123),
(timestamp("2023-12-06T12:00:00"), 123),
(timestamp("2023-12-07T12:00:00"), 123),
(timestamp("2023-12-08T12:00:00"), 123),
(timestamp("2023-12-09T12:00:00"), 123),
(timestamp("2023-12-10T12:00:00"), 123),
(timestamp("2023-12-11T12:00:00"), 123),
(timestamp("2023-12-12T12:00:00"), 123)

Francois Forster

12/13/2023, 8:09 PM

I add the freshness test:

Copy code

- elementary.freshness_anomalies:
          timestamp_column: "timestamp"
          days_back: 14
          backfill_days: 2
          time_bucket:
            period: day
            count: 1

Francois Forster

12/13/2023, 8:10 PM

When I run the report I see a dip in freshness before the spike if data is late:

Francois Forster

12/13/2023, 8:19 PM

And even when data isn't late there is the dip, which makes it alert all the time

Francois Forster

12/13/2023, 10:18 PM

Interestingly, I don't see the dip with backfill_days of 1

Francois Forster

12/13/2023, 10:31 PM

And if I choose backfill_days of 3, the dip is shifted

Ofek Weiss

01/08/2024, 8:42 AM

Hi @Francois Forster, sorry for the late response. i tried replicating this issue (according to your instructions) and everything seems to be working fine 🤔 do you still encounter this issue?

Francois Forster

01/08/2024, 2:56 PM

I still do

Ofek Weiss

01/09/2024, 10:50 AM

What version of

elementary

are you using? also, the

days_back

and

backfill_days

parameters were deprecated in favor of training_period and detection_period, mind trying to use those instead?

Francois Forster

01/09/2024, 3:00 PM

Copy code

dbt-core==1.6.7
dbt-bigquery==1.6.8
elementary-data[bigquery]== 0.11.0

Francois Forster

01/09/2024, 3:00 PM

Copy code

- package: elementary-data/elementary
    version: 0.12.1

Francois Forster

01/09/2024, 3:00 PM

I'll try to upgrade and switch the parameters

39 Views

Open in Slack

Previous Next