<@U088WCP3B4P> Hello, i have a volume test set on ...
# ask-elementary-ai
a
@kapa.ai Hello, i have a volume test set on a table that is basically recreated every day with new data. However when i check on the elementary dashboard, i see that the training period always has each day with 0 data except the last day that is out of training period. How can i make the volume test work in that case ?
k
A message from kapa.ai
For tables that are fully recreated/overwritten every day, timestamp‑based volume anomalies usually won’t work as you expect, because: - Elementary ignores the current (open) day’s bucket when using a
timestamp_column
, to avoid counting incomplete data. [[GA4 overwrite case](https://slack.com/archives/C02CTC89LAX/p1687424130703889)] - If all rows share the same “load” timestamp for that day, the metric for “yesterday” (the last closed bucket) can end up as 0, so your training period shows almost all zeros and only the latest day looks non‑zero / outside the training window. [[Overwrite models](https://slack.com/archives/C02CTC89LAX/p1687424130703889)] In overwrite scenarios like yours, the recommended pattern is to remove the timestamp column from the volume test, so Elementary tracks total table size per run instead of per day:
Copy code
yaml models:   - name: your_table     tests:       - elementary.volume_anomalies           # no timestamp_column here
With no
timestamp_column
: - Each test run records the entire table row count. - After you collect enough runs (default training period is ~14 days / runs), Elementary starts detecting anomalies in the total volume across those runs. [[Non‑timestamp behavior](https://slack.com/archives/C02CTC89LAX/p1650989426047689); [Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] This is exactly what Elementary support recommends for: - Tables that are truncated and reloaded. [[Truncate/reload sources](https://slack.com/archives/C02CTC89LAX/p1650989426047689)] - Tables overwritten daily where the load‑timestamp is the same for all rows. [[Overwrite models](https://slack.com/archives/C02CTC89LAX/p1687424130703889)] - Sources in overwrite mode whose “yesterday” bucket never contains the latest volume. [[Overwrite sources](https://slack.com/archives/C02CTC89LAX/p1710327533594939)] If you want the test to start reacting sooner than 14 days of history, you can lower the training size (e.g. via
min_training_set_size
), understanding that this trades off some statistical robustness. [[Training size behavior](https://slack.com/archives/C02CTC89LAX/p1690997004224019); [Training period docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period)] So, to fix your case: 1. Remove
timestamp_column
from the
elementary.volume_anomalies
config for that table. 2. Run
dbt test
on that table daily to accumulate training points. 3. Optionally tune training settings once you see enough points in
data_monitoring_metrics
.
a
@kapa.ai OK but the issue is that i use the timestamp_column also for a freshness_anomalies test like so :
Copy code
- name: visits_catalog_page_viewed
  description: "Raw catalog page viewed events from Oasis."
  config:
    elementary:
      timestamp_column: "tracking_timestamp"
  tests:
    - elementary.freshness_anomalies:
        timestamp_column: "tracking_timestamp"
        tags: [ "elementary", "monitoring_visits_catalog", "anomaly_detection" ]
        severity: warn
    - elementary.volume_anomalies:
        tags: [ "elementary", "monitoring_visits_catalog", "anomaly_detection" ]
        severity: warn
What can i do in this case ?
k
A message from kapa.ai
You’re running into this because the model‑level
timestamp_column
applies to both tests: once it’s configured under
config: elementary:
,
volume_anomalies
will also use it and bucket by day, which breaks for overwrite tables. [[Timestamp search order](https://slack.com/archives/C02CTC89LAX/p1681854412520199); [Timestamp config priority](https://slack.com/archives/C02CTC89LAX/p1681873690201069)] To have: -
freshness_anomalies
→ use
tracking_timestamp
-
volume_anomalies
→ ignore timestamps and use total table volume per run you need to remove the model‑level timestamp and configure it only on the freshness test, leaving the volume test without a
timestamp_column
. Example:
Copy code
yaml models:   - name: visits_catalog_page_viewed     description: "Raw catalog page viewed events from Oasis."     tests:       - elementary.freshness_anomalies:           timestamp_column: tracking_timestamp           tags: ["elementary", "monitoring_visits_catalog", "anomaly_detection"]           severity: warn        # total-volume anomalies, no timestamp -> snapshot per run       - elementary.volume_anomalies:           tags: ["elementary", "monitoring_visits_catalog", "anomaly_detection"]           severity: warn
Behavior with this setup: -
freshness_anomalies
must have a
timestamp_column
, which you now pass explicitly on the test. [[Freshness config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies)] -
volume_anomalies
has no timestamp, so it records the entire row count at each run and compares those snapshots over time; this is the recommended pattern for overwrite tables. [[No‑timestamp volume](https://slack.com/archives/C02CTC89LAX/p1710327533594939); [No‑timestamp behavior](https://slack.com/archives/C02CTC89LAX/p1695060702152949)] If you currently have
config.elementary.timestamp_column
defined globally (e.g., in
dbt_project.yml
), you’ll need to remove it from that model (or override it only where you truly want timestamp‑based volume).
a
@kapa.ai Actually in the code i provided before, the timestamp_column was redundant no ? Because it was set for all tests in that table + freshness
k
A message from kapa.ai
Yes, in that snippet the
timestamp_column
on the freshness test is redundant. You had:
Copy code
yaml - name: visits_catalog_page_viewed   config:     elementary:       timestamp_column: "tracking_timestamp"   tests:     - elementary.freshness_anomalies:         timestamp_column: "tracking_timestamp"
Elementary’s configuration resolution is: - You can set
timestamp_column
at model level (
config: elementary:
), - or at test level, - and for a single test these two are effectively equivalent. [[Config priority](https://slack.com/archives/C02CTC89LAX/p1681873690201069)] So:
Copy code
yaml config:   elementary:     timestamp_column: "tracking_timestamp"
already makes
freshness_anomalies
use `tracking_timestamp`; adding
timestamp_column
again under that test doesn’t change behavior. [[Timestamp column docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)] The only non‑redundant part is that by putting the timestamp at model level you also implicitly apply it to
volume_anomalies
, which is what caused the unwanted day‑bucket behavior for your overwrite table.