https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> Hello, so i have a test column_anom...
# ask-elementary-ai
a
@kapa.ai Hello, so i have a test column_anomalies that fails on some columns of my table on the min test specifically. I noticed that it fails because when the test runs, the data is incomplete which heavily affects the values tested on the min. So i was wondering if there's a way to test yesterday's data instead of the current day ?
k
A message from kapa.ai
Elementary’s anomaly detection tests, including column_anomalies, are designed to avoid exactly this issue: by default, they only evaluate “full” time buckets. For daily buckets, this means the test will only consider yesterday’s data, not today’s (which may be incomplete and cause false positives). So, if your test is configured with a timestamp column and the default daily time bucket, it should already be testing yesterday’s data, not the current day. If you are seeing failures due to incomplete data for today, it may indicate a configuration or data issue. Here are some key points: - By default, Elementary only calculates anomalies on completed time buckets, so for daily buckets, the latest bucket tested is yesterday, not today. If you run the test in the middle of today, it should only count yesterday as a complete bucket and ignore today’s partial data. This is intended to prevent exactly the kind of false positives you’re experiencing [volume_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). - If you have a timestamp column configured, make sure it is correctly set and that your data for yesterday is complete by the time the test runs [Anomaly troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). - If you are still seeing today’s (incomplete) data being tested, it could be a bug or a misconfiguration. In that case, double-check your test configuration and the timestamp column, and consider sharing your configuration for further troubleshooting [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1682044085432779). In summary: Elementary should, by default, test only yesterday’s data for daily buckets. If that’s not happening, please review your configuration and data, as this is not the expected behavior. Let me know if you’d like help reviewing your specific test configuration!
a
@kapa.ai OK let me add some more context. Today is the 25th of November 2025. First Here is the result of a BigQuery query where it shows that the min of column mpr has an abnormal value on the 2025-11-24 compared to the days before, because the rows_considered by the filter we apply is way less that day, mainly because we don't have enough data yet to test on it (result is in JSON) : [{ "day": "2025-11-24", "filtered_min_mpr": "11.06", "rows_considered": "4" }, { "day": "2025-11-23", "filtered_min_mpr": "6.0", "rows_considered": "94" }, { "day": "2025-11-22", "filtered_min_mpr": "5.35", "rows_considered": "79" }, { "day": "2025-11-21", "filtered_min_mpr": "5.37", "rows_considered": "75" }, { "day": "2025-11-20", "filtered_min_mpr": "4.88", "rows_considered": "75" }, { "day": "2025-11-19", "filtered_min_mpr": "4.68", "rows_considered": "73" }, { "day": "2025-11-18", "filtered_min_mpr": "6.42", "rows_considered": "72" }, { "day": "2025-11-17", "filtered_min_mpr": "5.2", "rows_considered": "72" }, { "day": "2025-11-16", "filtered_min_mpr": "3.62", "rows_considered": "85" }, { "day": "2025-11-15", "filtered_min_mpr": "5.49", "rows_considered": "64" }] So today as it is the 2025-11-25, i would actually like to test on the 2025-11-23 where we have a full day of data. On the elementary dashboard, it says this : "In column mpr, the last min value is 11.06. The average for this metric is 5.471." And here's the code of the test :
Copy code
- name: purchased_rank_stats
  description: "Metrics for purchased rank statistics."
  config:
    elementary:
      timestamp_column: "ranking_date"
  columns:
    - name: mpr
      description: "Mean purchased rank"
      tests:
        - elementary.column_anomalies:
            column_anomalies:
              - min
              - max
              - average
              - standard_deviation
            where_expression: >
              n_orders > 50 AND agora_variation NOT IN ('drk_popularity')
            tags: ["elementary", "monitoring_purchased_rank_stats", "anomaly_detection"]
            severity: warn
k
A message from kapa.ai
Thank you for the detailed context. Based on your description and the knowledge sources, here’s how Elementary handles this scenario and what you can do: ### How Elementary Determines Which Day to Test By default, Elementary’s anomaly detection tests (including 
column_anomalies
) only evaluate completed time buckets. For daily buckets, this means the test should only consider yesterday’s data (i.e., 2025-11-24 if today is 2025-11-25), not today’s partial data. This is designed to avoid false positives from incomplete data loads [volume_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). However, if your data for "yesterday" (2025-11-24) is still incomplete at the time of the test run, you may still get anomalies due to low row counts, as you observed. ### Your Specific Need: Test on the Last Fully Loaded Day Currently, Elementary does not natively support skipping the most recent day and testing the last fully loaded day (e.g., 2025-11-23 in your case) if yesterday’s data is incomplete. The system always tests the most recent completed bucket based on the timestamp column and the time the test is run. If the data for that bucket is incomplete, it can lead to the kind of false positives you’re seeing. This limitation is a known issue and has been discussed in the community. There have been feature requests to allow for a configurable delay or to skip the most recent X days, but as of now, this is not available out-of-the-box [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1680016519796649). ### Workarounds - Delay Test Runs: The recommended workaround is to schedule your dbt/Elementary test runs to occur only after you are confident that the previous day’s data is fully loaded. This ensures that the “yesterday” bucket is complete when the test runs [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671002553673119). - Custom Filtering: You could add a 
where_expression
to further restrict the data Elementary tests on, but this won’t dynamically skip incomplete days—it will only filter rows within the selected bucket. - Feature Request: If this is a recurring issue, consider upvoting or contributing to the relevant feature request on GitHub, as the Elementary team is aware of this need [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1680016519796649). ### Summary - Elementary tests only completed buckets (usually yesterday), but if yesterday’s data is incomplete, it will still be tested and may cause false positives. - There is no built-in way to automatically skip to the last fully loaded day (e.g., two days ago) if yesterday is incomplete. - Best practice: Schedule your tests after data loads are complete, or monitor for future updates from Elementary that may address this feature. If you need help with scheduling or want to discuss a more advanced workaround, let me know!
a
@kapa.ai Would it work if a do something like that : timestamp_column: "DATE_ADD(ranking_date, INTERVAL 1 DAY)"
k
A message from kapa.ai
Based on the available knowledge sources, there is no explicit documentation or community confirmation that using an expression like 
timestamp_column: "DATE_ADD(ranking_date, INTERVAL 1 DAY)"
is officially supported in Elementary’s test configuration. The documentation and Slack discussions indicate that the 
timestamp_column
parameter expects a column name, and while there have been user requests to support calculated fields or expressions, this is not currently a documented feature [Slack discussion](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683764020831119). A collaborator suggested the idea of supporting calculated fields or a 
convert_timezone
config, but as of now, this is not implemented. The recommended approach remains to use a column name for 
timestamp_column
and, if needed, create a new column in your model or staging layer that contains the adjusted date, then reference that column in your test configuration. In summary: Using an expression like 
DATE_ADD(ranking_date, INTERVAL 1 DAY)
directly in the 
timestamp_column
parameter is not officially supported according to the current knowledge sources. If you need this logic, you should create a new column in your model with the adjusted date and use that column as your 
timestamp_column
 [Slack discussion](https://elementary-community.slack.com/archives/C02CTC89LAX/p1683764020831119).
36 Views
Open in Slack
PreviousNext
Powered by