<@U088WCP3B4P> proddbt_shobia_common_refined4item_...
# ask-elementary-ai
s
@kapa.ai proddbt_shobia_common_refined4item_branch__parquet_common_refined4itembranch.creation dateanomaly_detectiongenericDatabase Error in test elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3 (models\refined4\common\quality_check\_source_quality_check_common_refined4.yml) syntax error at or near "from" in context "as start_bucket_in_data, from", at line 26, column 13[]["common_refined4", "daily_elementary", "elementary", "elementary-tests", "refined4"]elementary_column_anomalies_item_branch__parquet_common_refined4_both__2__null_count__itembranch_creation_date__3__3{"column_anomalies": ["null_count"], "anomaly_sensitivity": 2, "anomaly_direction": "both", "ignore_small_changes": {"spike_failure_percent_threshold": 3, "drop_failure_percent_threshold": 3}, "column_name": "itembranch.creation date", "model": "{{ get_where_subquery(ref('item_branch__parquet_common_refined4')) }}"} why it is saying database error i just define dthe test without timestamp column for the the column name "itembranch.creation date" to check the null count anomoly like yesterday this much today there is a spike so detect anomoly . how to define that - name: "itembranch.creation date" tests: - elementary.column_anomalies: column_anomalies: - null_count anomaly_sensitivity: 2 anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 3 drop_failure_percent_threshold: 3 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
k
A message from kapa.ai
You’re hitting a known Elementary bug that appears when
column_anomalies
(and sometimes
volume_anomalies
) run without a
timestamp_column
: the generated SQL includes an extra comma in
... as start_bucket_in_data, from ...
, which causes a syntax error in several warehouses (Redshift, Databricks, Trino). [[Redshift bug](https://github.com/elementary-data/elementary/issues/1540); [Databricks bug](https://github.com/elementary-data/elementary/issues/1554)] In your error: >
syntax error at or near "from" in context "as start_bucket_in_data, from"
that’s exactly the same pattern as described in the open GitHub issues. [[Redshift bug](https://github.com/elementary-data/elementary/issues/1540); [Databricks bug](https://github.com/elementary-data/elementary/issues/1554)] ### Why it happens When no
timestamp_column
is configured, Elementary should monitor the full table over runs (no time bucketing), which is a fully supported mode. [[Column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies); [Timestamp behavior](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)] But in affected versions (e.g. package
0.15.2
), the macro that builds the monitoring query has a trailing comma in the
SELECT
when
timestamp_column
is missing, producing SQL like:
Copy code
sql select <your_column>, {{ elementary.null_timestamp() }} as start_bucket_in_data,  from monitored_table
This is exactly what the GitHub issues report. [[Redshift bug](https://github.com/elementary-data/elementary/issues/1540); [Databricks bug](https://github.com/elementary-data/elementary/issues/1554)] ### How to fix / work around it You have two options, depending on whether you can use a timestamp: #### 1. Recommended: add
timestamp_column
(workaround and better behavior) If your model has a creation/loaded timestamp, configure it so Elementary buckets by time and avoids the broken code path:
Copy code
yml models:   - name: item_branch__parquet_common_refined4     config:       elementary:         timestamp_column: itembranch_creation_date   # or another appropriate timestamp column     columns:       - name: "itembranch.creation date"         tests:           - elementary.column_anomalies:               column_anomalies:                 - null_count               anomaly_sensitivity: 2               anomaly_direction: both               ignore_small_changes:                 spike_failure_percent_threshold: 3                 drop_failure_percent_threshold: 3               tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
With `timestamp_column`: - Elementary splits data into time buckets using that column. - It compares
null_count
across buckets (e.g. yesterday vs previous days) to detect spikes/drops. [[Timestamp behavior](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column); [Column anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies#test-configuration)] This is the same workaround other users used to bypass the bug. [[Databricks bug](https://slack.com/archives/C02CTC89LAX/p1716306300184349)] #### 2. If you truly cannot use a timestamp The docs say tests should work without a
timestamp_column
and just compare metrics between test runs. [[Timestamp behavior](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column); [Column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies)] However, the current open issues show that in some versions/warehouses this path is broken by the extra comma. [[Redshift bug](https://github.com/elementary-data/elementary/issues/1540); [Databricks bug](https://github.com/elementary-data/elementary/issues/1554)] Since the knowledge sources do not state which exact version contains the fix, I cannot confidently say “upgrade to version X and it will be solved.” The maintainers mention a PR was merged “in the near future,” but no tagged version is given. [[1554 comments](https://github.com/elementary-data/elementary/issues/1554)] Given that, your realistic options are: - Temporarily add a suitable timestamp column (best and simplest workaround). - Or, if you know you’re on a version with this bug (e.g.
0.15.2
), upgrade Elementary dbt package / CLI to the latest available in your environment and re-run; if the issue persists, it’s still the same open bug. ### Why your current config fails Your test (without a timestamp):
Copy code
yml - name: "itembranch.creation date"   tests:     - elementary.column_anomalies:         column_anomalies:           - null_count         anomaly_sensitivity: 2         anomaly_direction: both         ignore_small_changes:           spike_failure_percent_threshold: 3           drop_failure_percent_threshold: 3         tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
is conceptually correct and matches the documented pattern for running
column_anomalies
without a
timestamp_column
. [[Column anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies)] The database error is not because of your YAML, but because of the known SQL generation bug on the no-timestamp path. --- Summary - Your YAML is valid; the error is from a known bug in the generated SQL when
timestamp_column
is omitted. - To detect “yesterday vs today” spikes in
null_count
, configure a
timestamp_column
for the model and keep your test as is. - If you cannot use a timestamp, you’re blocked by this bug until you’re on a version where that PR is released; the knowledge sources don’t specify which version that is.