Winston users would like data quality checks on "C...
# ask-elementary-ai
s
Winston users would like data quality checks on "Creation Date" Field in Item Branch Description Add and Elementary data anomaly checks to the parquet refined4 common refined4 ItemBranch • for rows that have a value can we check changes in the average date age? • the number of records that have dates over time • check if there are nulls, I think this is likely - name: item_branch__parquet_common_refined4 data_tests: - elementary.volume_anomalies: anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] columns: - name: "%itembranchskey" quote: true data_tests: - unique: config: severity: error error_if: ">1" warn_if: ">1" - not_null add along with this the field is "itembranch.creation date"
k
A message from kapa.ai
To add Elementary data anomaly checks on the "Creation Date" field in your
item_branch__parquet_common_refined4
model, you can use the following Elementary tests based on your requirements: ### 1. Check for changes in the average date age You can use the
elementary.column_anomalies
test on the "creation date" column and specify the
average
monitor. This will track the average value of the date (interpreted as a numeric value, e.g., days since epoch) over time and alert on anomalies. ### 2. Monitor the number of records with dates over time Use the
elementary.volume_anomalies
test with
timestamp_column
set to your "creation date" field. This will monitor the row count per time bucket (e.g., per day) for records that have a creation date. ### 3. Check for nulls in the "creation date" field Add a standard
not_null
test on the "creation date" column. --- Below is an example configuration you can add to your dbt model YAML file:
Copy code
yaml models:   - name: item_branch__parquet_common_refined4     config:       elementary:         timestamp_column: creation_date     tests:       - elementary.volume_anomalies:           timestamp_column: creation_date           anomaly_direction: both           ignore_small_changes:             spike_failure_percent_threshold: 1             drop_failure_percent_threshold: 1           tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]     columns:       - name: "%itembranchskey"         quote: true         tests:           - unique:               config:                 severity: error                 error_if: ">1"                 warn_if: ">1"           - not_null       - name: creation_date         tests:           - not_null           - elementary.column_anomalies:               column_anomalies:                 - average               tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
Notes: - The
timestamp_column
should be set to your "creation date" field to enable time-bucketed anomaly detection for both volume and column metrics. This is highly recommended for accurate and timely anomaly detection [see details](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column). - The
average
monitor on a date field will track the average value (e.g., average days since epoch). If you want to track a different metric, you can adjust the
column_anomalies
list accordingly [see supported metrics](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/column-anomalies). - The
not_null
test will alert if any nulls are present in the "creation date" field. This configuration will provide the checks you described, leveraging Elementary's anomaly detection and dbt's standard data quality tests. References: - [Column anomalies configuration](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/column-anomalies) - [Volume anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies) - [Timestamp column usage](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)