- name: "itembranch.creation date" tests: ...
# ask-elementary-ai
s
- name: "itembranch.creation date" tests: - elementary.column_anomalies: column_anomalies: - null_count timestamp_column: null anomaly_sensitivity: 2 anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 3 drop_failure_percent_threshold: 3 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] Elementary: Not enough data to calculate anomaly scores on
test.pem_dbt.elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3__None.77313c61c8
k
A message from kapa.ai
You’re seeing the expected Elementary behavior: there isn’t enough training data yet for this
column_anomalies
test, so Elementary can’t compute an anomaly score and prints: > Not enough data to calculate anomaly scores … Key points from the docs and Slack threads: 1. Timestamp vs. no timestamp - If you configure a
timestamp_column
, Elementary buckets data by time and can often start detecting anomalies immediately, as long as there is enough historical data in those buckets.[[Anomaly params](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-params); [Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] - If you do not have a timestamp (or set it to
null
as in your config), Elementary treats each test run as a time point; you must accumulate enough runs over different days for a training set.[[Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] In your YAML you have:
Copy code
yaml    timestamp_column: null
which effectively configures this as a non‑timestamp test. 2. How much data is “enough”? There are two slightly different thresholds in the material you shared: - Some docs/issues mention a default
min_training_set_size
of 14
(14 days / runs).[[Slack 1682967620](https://slack.com/archives/C02CTC89LAX/p1682967620411319); [Slack 1692691883](https://slack.com/archives/C02CTC89LAX/p1692691883697119)] - The Troubleshooting guide describes “Not enough data to calculate anomaly” as fewer than 7 training data points.[[Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] Either way, the message means: Elementary has fewer than the required number of historical points for
null_count
on this column. 3. What you can do Option A – Use a real timestamp column (recommended) If your table has a creation or partition date column, configure it as the table‑level timestamp instead of putting
timestamp_column: null
in the test:
Copy code
yaml    models:      - name: item_branch__parquet        config:          elementary:            timestamp_column: creation_date   # or the correct time column        columns:          - name: "itembranch.creation date"            tests:              - elementary.column_anomalies:                  column_anomalies:                    - null_count                  anomaly_sensitivity: 2                  anomaly_direction: both                  ignore_small_changes:                    spike_failure_percent_threshold: 3                    drop_failure_percent_threshold: 3                  tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
With a proper timestamp and enough historical data in that column, Elementary can often calculate anomaly scores immediately.[[Column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies); [Configure tests](https://docs.elementary-data.com/data-tests/add-elementary-tests#configure-your-elementary-anomaly-detection-tests)] Option B – Keep it non‑timestamp and build training data If you really want to run without a timestamp: - Keep
timestamp_column
out of the test (don’t set it to
null
at the test level; just omit it, as in the non‑timestamp example in the docs).[[Column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies)] - Run the test on multiple different days so that
data_monitoring_metrics
accumulates enough
null_count
points for this column.[[Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide); [Slack 1687786556](https://slack.com/archives/C02CTC89LAX/p1687786556886149)] - Optionally lower
min_training_set_size
globally if you only need a demo or are okay with less stable thresholds (the exact YAML for that isn’t in the snippets you provided, only that it exists).[[Slack 1700704884](https://slack.com/archives/C02CTC89LAX/p1700704884937269); [Slack 1692691883](https://slack.com/archives/C02CTC89LAX/p1692691883697119)] 4. How to verify Use the troubleshooting queries to confirm metrics are being collected:
Copy code
sql    SELECT        metric_name,        metric_timestamp,        metric_value    FROM your_schema.data_monitoring_metrics    WHERE table_name = 'item_branch__parquet'      AND column_name = 'itembranch.creation date'    ORDER BY metric_timestamp DESC;
You should see multiple
null_count
rows over different timestamps; if there are fewer than the training minimum, you’ll keep getting the “Not enough data” message.[[Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] --- In short: the message is expected until you either (a) provide a real timestamp column with enough history, or (b) run the non‑timestamp test enough times (or with a lower
min_training_set_size
) to build a training set.