@kapa.ai Subject: Handling

volume_anomaly

and

freshness_anomaly

tests for business-day-only Airflow orchestration Question: Is it possible to implement Elementary's

volume_anomaly

and

freshness_anomaly

tests on a dbt model that is orchestrated via Airflow with a business-day-only schedule (12 PM)? The logic of the pipeline is to process all "pending" data since the last successful run. This creates an uneven distribution of data per execution: • Monday Run: Ingests data from Friday, Saturday, and Sunday. • Holiday Scenario: If Friday was a holiday, the Monday run ingests data from Thursday, Friday, Saturday, and Sunday. • Tuesday Run: Ingests only Monday's data. Considering the Elementary documentation for these tests: 1. Can these ingestion "bursts" be handled using the standard dbt YAML syntax for Elementary tests? 2. How should I configure the

time_bucket

or the

event_timestamp

to avoid false positives on Mondays (volume spikes) or over the weekend (freshness delays)? 3. Or, given this specific business logic, is it necessary to design a custom test instead?

Elementary can support this kind of “bursty” business‑day ingestion, but you’ll need to be deliberate about how you use

timestamp_column

/

time_bucket

, and you may want to combine multiple tests rather than rely on a single configuration. Below I’ll separate guidance for

volume_anomalies

and

freshness_anomalies

, and then address whether you really need custom tests. ------------------------- 1. Volume anomalies for business‑day bursts Option A – Treat each run as a snapshot (no timestamp) If your Airflow DAG runs only on business days and each run ingests “all pending data since last success”, the simplest and most robust pattern is: • Do not use

timestamp_column

on the

volume_anomalies

test. • Let Elementary record the total row count per run and compare those snapshots over time. [Weekly load example; Overwrite-table guidance] This way: • Monday’s “big” run (Fri+Sat+Sun, or Thu–Sun on holidays) is just “a typical Monday snapshot” and will be learned as such. • Tuesday’s smaller run (only Monday) is “a typical Tuesday snapshot”. • You’re not forcing Elementary to reason about per‑day volume; it just learns the pattern of per‑run volume. YAML example:

models:
  - name: your_model
    tests:
      # Snapshot-per-run volume anomalies
      - elementary.volume_anomalies:
          # no timestamp_column here
          # optional: seasonality by day_of_week if you want Mon vs Tue patterns
          seasonality: day_of_week

This is the same “no timestamp” mode recommended for weekly loads and overwrite tables. [Weekly load example; No‑timestamp volume] Option B – Use

timestamp_column

+

time_bucket

only if you truly need per‑day volume If you must detect anomalies per event day (e.g., “Sunday volume is off”), you can: • Set

timestamp_column

to the event date. • Use

time_bucket

with

period: day, count: 1

. [time_bucket docs] • Optionally add

seasonality: day_of_week

so Mondays are compared to Mondays, etc. [seasonality docs] However: • Elementary only evaluates *full buckets*; with daily buckets, that means “up to yesterday”. [Full‑day behavior; Day‑of‑week + full day requirement] • It still expects each calendar day’s volume to be comparable to previous same‑weekday days; your “bursty” ingestion is fine as long as the event timestamps reflect the true event day and the pattern is stable. Given your description, Option A (snapshot per run) is usually more stable and avoids Monday “spike” false positives. ------------------------- 2. Freshness anomalies with business‑day ingestion There are two different “freshness” needs that show up in your scenario: 1. “Did we run on time today?” (pipeline recency) 2. “How old are the events we just ingested?” (event‑to‑now lag) Elementary has multiple ways to handle this, but the knowledge sources don’t fully spell out all configuration details for

freshness_anomalies

in this exact pattern. I’ll stick to what is documented. A. Use standard dbt freshness tests for “did data arrive?” Several users combine Elementary with dbt’s built‑in freshness tests to answer “did the table get updated when expected?” and then let Elementary handle anomalies on other metrics. [Freshness combo discussion] In your case: • dbt freshness can be configured to expect updates only on business days. • Elementary will still surface alerts for those dbt tests, even though they’re not Elementary tests per se. [Freshness combo discussion] This avoids weekend “false delays” because you simply don’t expect runs on weekends. B. Event‑level freshness anomalies (lag between event time and now) Elementary added

event_freshness_anomalies

specifically for cases where loading happens later than the event time, and you want to measure the lag between event timestamp and now. [event_freshness_anomalies intro] This is a good fit if: • Events from Fri–Sun are all loaded on Monday, but you want to detect if that lag suddenly grows (e.g., you start loading events 5 days late instead of 2–3). • You care about “event recency” rather than “table last updated at”. The docs link in the thread is generic, but the intent is clear: this test is designed for delayed ingestion patterns like yours. [event_freshness_anomalies intro] ------------------------- 3. Handling weekends / holidays and avoiding false positives From the knowledge sources: • Seasonality (

day_of_week

,

hour_of_day

,

hour_of_week

) is the main built‑in way to handle recurring patterns like weekends. [seasonality docs] • There is no built‑in way to “skip” specific days (e.g., weekends) from detection while still using

fail_on_zero

for weekdays; this is a known limitation. [Weekend exclusion limitation] • For cases where you don’t want to evaluate the very latest data (e.g., partial loads), you can use

detection_delay

to exclude the most recent period from detection. [detection_delay suggestion] Applied to your setup: • If you use snapshot‑per‑run volume anomalies (no timestamp), you don’t need to worry about weekends at all; there are simply no runs on those days. • For event‑based tests (volume or freshness with

timestamp_column

), use: ◦

seasonality: day_of_week

if you want Monday patterns vs Tuesday patterns. ◦

detection_delay

if you need to ignore the most recent partial period (less relevant if you only run once per business day at noon). ------------------------- 4. Do you need a custom test? Based on the knowledge sources, you can stay within standard dbt YAML + Elementary tests: •

volume_anomalies

without

timestamp_column

to learn per‑run volume (handles Monday bursts and holiday shifts). [Weekly load example; No‑timestamp volume] • dbt freshness tests for “did we run on time on business days?”. [Freshness combo discussion] •

event_freshness_anomalies

if you also want anomaly detection on event‑to‑now lag. [event_freshness_anomalies intro] The knowledge sources do not indicate that a custom test is required for your pattern; they instead show how similar “non‑daily / bursty” or “weekly” ingestion patterns are handled via: • No‑timestamp mode •

time_bucket

•

seasonality

•

detection_delay

• Combining Elementary with dbt’s own freshness tests If you find that you must exclude specific calendar days from detection while still using

fail_on_zero

and timestamp‑based buckets, that’s where current capabilities hit a limitation and a custom approach (or upstream modeling change) might be needed. [Weekend exclusion limitation] ------------------------- Recommended minimal setup for your case 1. Volume anomalies (per run):

models:
  - name: your_model
    tests:
      - elementary.volume_anomalies:
          seasonality: day_of_week   # optional but helpful

2. Freshness (pipeline recency): • Use dbt’s built‑in freshness test on the model or source, configured for business‑day expectations; Elementary will still alert on it. [Freshness combo discussion] 3. Optional: event‑lag anomalies:

models:
  - name: your_model
    tests:
      - elementary.event_freshness_anomalies:
          timestamp_column: event_timestamp  # your event time column

This combination should handle your business‑day‑only Airflow orchestration without needing custom tests. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.