https://linen.dev logo
Join Slack
Powered by
PS V:\pem_dbt> dbt test -m item_branch__parquet...
# ask-elementary-ai
s
PS V:\pem_dbt> dbt test -m item_branch__parquet_common_refined4 064130 Running with dbt=1.8.3 064131 Registered adapter: redshift=1.8.1 064132 [WARNING]: Deprecated functionality The 
tests
config has been renamed to 
data_tests
. Please see https://docs.getdbt.com/docs/build/data-tests#new-data_tests-syntax for more information. 064134 Found 759 models, 4 seeds, 4 operations, 746 data tests, 460 sources, 1327 macros 064134 064144 064144 Running 2 on-run-start hooks 064144 INFO: macro on_run_start skipped because schema not dbt_main, dbt_uat or dbt_prod 064144 1 of 2 START hook: pem_dbt.on-run-start.0 ...................................... [RUN] 064144 1 of 2 OK hook: pem_dbt.on-run-start.0 ......................................... [OK in 0.00s] 064145 IMPORTANT - Starting from dbt 1.8, users must explicitly allow packages to override materializations. Elementary requires this ability to support collection of samples and failed row count for dbt tests. Please add the following flag to dbt_project.yml to allow it: flags: require_explicit_package_overrides_for_builtin_materializations: false Notes - * This is a temporary measure that will result in a deprecation warning, please ignore it for now. Elementary is working with the dbt-core team on a more permanent solution. * This message can be muted by setting the 'mute_ensure_materialization_override' var to true. 064145 2 of 2 START hook: elementary.on-run-start.0 ................................... [RUN] 064145 2 of 2 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s] 064145 064145 Concurrency: 10 threads (target='main') 064145 064145 1 of 5 START test elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3__None [RUN] 064145 2 of 5 START test elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_percent__itembranch_creation_date__3__3__current_date [RUN] 064145 3 of 5 START test elementary_volume_anomalies_item_branch__parquet_common_refined4_both__1__1 [RUN] 064145 4 of 5 START test not_null_item_branch__parquet_common_refined4__itembranchskey_ [RUN] 064145 5 of 5 START test unique_item_branch__parquet_common_refined4__itembranchskey_ . [RUN] 064145 Elementary: Started running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 064147 4 of 5 PASS not_null_item_branch__parquet_common_refined4__itembranchskey_ ..... [PASS in 2.54s] 064151 Elementary: Finished running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 064152 3 of 5 PASS elementary_volume_anomalies_item_branch__parquet_common_refined4_both__1__1 [PASS in 7.21s] 064153 5 of 5 PASS unique_item_branch__parquet_common_refined4__itembranchskey_ ....... [PASS in 7.50s] 064153 Elementary: Started running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 itembranch.creation date 064153 Elementary: Started running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 itembranch.creation date 064200 Elementary: Finished running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 itembranch.creation date 064200 2 of 5 PASS elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_percent__itembranch_creation_date__3__3__current_date [PASS in 15.73s] 064223 Elementary: Finished running data monitors on: PROD.DBT_SHOBIA_COMMON_REFINED4.ITEM_BRANCH__PARQUET_COMMON_REFINED4 itembranch.creation date 064223 Elementary: Not enough data to calculate anomaly scores on 
test.pem_dbt.elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3__None.77313c61c8
064224 1 of 5 PASS elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3__None [PASS in 38.93s] 064224 064224 Running 2 on-run-end hooks 064224 INFO: macro on_run_end skipped because schema not dbt_main, dbt_uat or dbt_prod 064224 1 of 2 START hook: pem_dbt.on-run-end.0 ........................................ [RUN] 064224 1 of 2 OK hook: pem_dbt.on-run-end.0 ........................................... [OK in 0.00s] - name: item_branch__parquet_common_refined4 tests: - elementary.volume_anomalies: anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] columns: - name: "%itembranchskey" quote: true data_tests: - unique: config: severity: error error_if: ">1" warn_if: ">1" - not_null - name: "itembranch.creation date" tests: - elementary.column_anomalies: column_anomalies: - null_count timestamp_column: null anomaly_sensitivity: 2 anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 3 drop_failure_percent_threshold: 3 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] - name: "itembranch.creation date" tests: - elementary.column_anomalies: column_anomalies: - null_percent timestamp_column: current_date anomaly_sensitivity: 2 anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 3 drop_failure_percent_threshold: 3 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] when i give timestamp column: null then it is showning not enough data?
k
A message from kapa.ai
You’re seeing the expected behavior for a non‑timestamp anomaly test. ### Why 
timestamp_column: null
gives “Not enough data…” When you don’t configure a real 
timestamp_column
, Elementary treats the test as non‑timestamp‑based: - Each run computes the metric (here 
null_count
) over the entire table. - The “timestamp” for training is effectively the test run time (
updated_at
in 
data_monitoring_metrics
). [[timestamp column](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)] - Elementary needs a minimum number of historical points before it can calculate an anomaly score. The troubleshooting guide says anomalies aren’t calculated when there are fewer than 7 training data points. [[Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] So with:
Copy code
yaml - elementary.column_anomalies:     column_anomalies:       - null_count     timestamp_column: null
you must: 1. Run this test multiple times on different days (at least 7 runs; in practice many setups use 14 days as the min training set size). [[Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide); [Slack – no timestamp](https://slack.com/archives/C02CTC89LAX/p1692691883697119)] 2. Confirm that metrics are being collected by querying 
data_monitoring_metrics
(you should see one 
null_count
metric per run for that column). [[Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] Until there are enough historical rows for that specific metric ID, Elementary will log: > Not enough data to calculate anomaly scores … This is independent of your 
anomaly_sensitivity
etc. ### Why the other test with 
timestamp_column: current_date
behaves differently For the second test:
Copy code
yaml - elementary.column_anomalies:     column_anomalies:       - null_percent     timestamp_column: current_date
you are giving a (somewhat unusual) timestamp expression, so it becomes a timestamp‑based test. With a timestamp column and enough historical data in the table, Elementary can often calculate anomalies immediately because it can build the training set from existing historical buckets in a single run. [[timestamp column](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)] ### What you can do - If you want non‑timestamp behavior (compare run‑to‑run totals): - Keep 
timestamp_column
unset or 
null
. - Run this test daily until you have enough history (check 
data_monitoring_metrics
to confirm multiple rows for 
null_count
for that column). [[Anomaly troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide)] - If your table has a real date/timestamp column (e.g. 
itembranch.creation date
) and you want faster results: - Configure that real column as the 
timestamp_column
(model‑ or test‑level) so Elementary can bucket by that field and use existing history. [[column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies#test-configuration); [timestamp column](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)] If you share how many rows you see in 
data_monitoring_metrics
for this 
null_count
test, I can help you interpret whether you’ve crossed the training threshold.
33 Views
Open in Slack
PreviousNext
Powered by