<@U088WCP3B4P> Winston users would like data quali...
# ask-elementary-ai
s
@kapa.ai Winston users would like data quality checks on "Creation Date" Field in Item Branch Description Add and Elementary data anomaly checks to the parquet refined4 common refined4 ItemBranch • for rows that have a value can we check changes in the average date age? • the number of records that have dates over time • check if there are nulls, I think this is likely - name: item_branch__parquet_common_refined4 data_tests: - elementary.volume_anomalies: anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] columns: - name: "%itembranchskey" quote: true data_tests: - unique: config: severity: error error_if: ">1" warn_if: ">1" - not_null add along with this the field is "itembranch.creation date" @kapa.ai how will the avergae works for a date column
k
A message from kapa.ai
To address your requirements for data quality checks on the "Creation Date" field in the
item_branch__parquet_common_refined4
model using Elementary, here’s how you can approach each check: ### 1. Changes in the Average Date Age To monitor the "average date age" (i.e., the average difference between the current date and the "Creation Date"), you can use the
elementary.column_anomalies
test with the
average
monitor. For a date column, the "average" will be calculated as the average of the date values, which typically results in a timestamp or date. To get the "age," you would need to create a derived column (e.g.,
DATEDIFF(current_date, creation_date)
) and monitor its average. However, Elementary's built-in
average
monitor on a date column will just compute the average date, not the age in days. If you want to specifically monitor the age, you should create a new column in your model for this calculation and then apply the anomaly test to that column. ### 2. Number of Records with Dates Over Time You can use the
elementary.volume_anomalies
test with the
timestamp_column
set to "creation date." This will monitor the count of records per time bucket (e.g., daily) based on the "creation date" field, allowing you to detect spikes or drops in the number of records created over time. This is a recommended approach for tracking volume changes related to a date field [Volume anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). ### 3. Null Checks on "Creation Date" To check for nulls in the "creation date" field, you can use the
elementary.column_anomalies
test with the
null_count
or
null_percent
monitor. This will alert you if there is an unexpected increase in null values for this column [Column anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies). --- ### Example YAML Configuration
Copy code
yaml models:   - name: item_branch__parquet_common_refined4     tests:       - elementary.volume_anomalies:           timestamp_column: creation_date           anomaly_direction: both           ignore_small_changes:             spike_failure_percent_threshold: 1             drop_failure_percent_threshold: 1           tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]     columns:       - name: creation_date         tests:           - elementary.column_anomalies:               column_anomalies:                 - null_count                 - average       - name: "%itembranchskey"         quote: true         tests:           - unique:               config:                 severity: error                 error_if: ">1"                 warn_if: ">1"           - not_null
Note: - The
average
monitor on a date column will compute the average date, not the average age. For average age, create a derived column (e.g.,
age_in_days
) and monitor that. - Setting
timestamp_column: creation_date
in
volume_anomalies
will bucket your data by creation date, which is the recommended approach for time-based anomaly detection [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column). If you need to monitor the average age specifically, you should add a column to your model like:
Copy code
sql SELECT   *,   DATEDIFF(current_date, creation_date) AS creation_date_age FROM ...
Then, add an anomaly test on
creation_date_age
with the
average
monitor. --- Summary: - Use
elementary.volume_anomalies
with
timestamp_column: creation_date
for volume over time. - Use
elementary.column_anomalies
with
null_count
for null checks. - Use
elementary.column_anomalies
with
average
for average date (or create an age column for average age). - For the average of a date column, Elementary computes the mean date, not the age; for age, use a derived column. References: [Column anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies) [Volume anomalies test](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies) [timestamp_column documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/timestamp-column)