shobia
11/06/2025, 9:38 AMkapa.ai
11/06/2025, 9:39 AMkapa.ai
11/06/2025, 9:39 AMitem_branch__parquet_common_refined4 model using Elementary, here’s how you can approach each check: ### 1. Changes in the Average Date Age To monitor the "average date age" (i.e., the average difference between the current date and the "Creation Date"), you can use the elementary.column_anomalies test with the average monitor. For a date column, the "average" will be calculated as the average of the date values, which typically results in a timestamp or date. To get the "age," you would need to create a derived column (e.g., DATEDIFF(current_date, creation_date)) and monitor its average. However, Elementary's built-in average monitor on a date column will just compute the average date, not the age in days. If you want to specifically monitor the age, you should create a new column in your model for this calculation and then apply the anomaly test to that column. ### 2. Number of Records with Dates Over Time You can use the elementary.volume_anomalies test with the timestamp_column set to "creation date." This will monitor the count of records per time bucket (e.g., daily) based on the "creation date" field, allowing you to detect spikes or drops in the number of records created over time. This is a recommended approach for tracking volume changes related to a date field [Volume anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). ### 3. Null Checks on "Creation Date" To check for nulls in the "creation date" field, you can use the elementary.column_anomalies test with the null_count or null_percent monitor. This will alert you if there is an unexpected increase in null values for this column [Column anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies). --- ### Example YAML Configuration yaml models: - name: item_branch__parquet_common_refined4 tests: - elementary.volume_anomalies: timestamp_column: creation_date anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] columns: - name: creation_date tests: - elementary.column_anomalies: column_anomalies: - null_count - average - name: "%itembranchskey" quote: true tests: - unique: config: severity: error error_if: ">1" warn_if: ">1" - not_null Note: - The average monitor on a date column will compute the average date, not the average age. For average age, create a derived column (e.g., age_in_days) and monitor that. - Setting timestamp_column: creation_date in volume_anomalies will bucket your data by creation date, which is the recommended approach for time-based anomaly detection [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column). If you need to monitor the average age specifically, you should add a column to your model like: sql SELECT *, DATEDIFF(current_date, creation_date) AS creation_date_age FROM ... Then, add an anomaly test on creation_date_age with the average monitor. --- Summary: - Use elementary.volume_anomalies with timestamp_column: creation_date for volume over time. - Use elementary.column_anomalies with null_count for null checks. - Use elementary.column_anomalies with average for average date (or create an age column for average age). - For the average of a date column, Elementary computes the mean date, not the age; for age, use a derived column. References: [Column anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies) [Volume anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies) [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)