<@U088WCP3B4P> name: item_branch__parquet_common_r...
# ask-elementary-ai
s
@kapa.ai name: item_branch__parquet_common_refined4 data_tests: - elementary.volume_anomalies: anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] what does this test do, if it does anything i want this also and i also have an concern does average column anomalies how ti works , give an example for this
k
A message from kapa.ai
Your configuration for the model item_branch__parquet_common_refined4:
Copy code
yaml data_tests:   - elementary.volume_anomalies:       anomaly_direction: both       ignore_small_changes:         spike_failure_percent_threshold: 1         drop_failure_percent_threshold: 1       tags: ["elementary", "daily_elementary", "refined4", "common_refined4"]
### What does this test do? - elementary.volume_anomalies monitors the row count of your table over time, split into time buckets (daily by default). - anomaly_direction: both means the test will flag anomalies for both unexpected spikes (increases) and drops (decreases) in row count. - ignore_small_changes with both thresholds set to 1 means the test will only fail if the row count in the detection period is more than 1% higher (spike) or 1% lower (drop) than the mean value in the training period, in addition to being statistically anomalous. This helps filter out minor fluctuations that are not meaningful. - tags are just metadata for organizing and filtering tests. So, this test will alert you if there is a statistically significant and meaningful (greater than 1% change) spike or drop in the number of rows in your table, based on recent historical data. This is useful for catching unexpected data volume changes, such as missing data or duplicate loads, while ignoring very small, insignificant changes [ignore_small_changes](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes) [volume_anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/volume-anomalies). --- ### How does the "average" column anomaly work? The average column anomaly monitor checks for anomalies in the average value of a numeric column over time. It works by: - Calculating the average value of the column for each time bucket (e.g., daily). - Comparing the average in the detection period (e.g., last 2 days) to the historical averages in the training period (e.g., previous 14 days). - If the average in the detection period is an outlier compared to the expected range (based on historical data), it is flagged as an anomaly. #### Example configuration:
Copy code
yaml models:   - name: your_model     columns:       - name: your_numeric_column         tests:           - elementary.column_anomalies:               column_anomalies:                 - average
This will monitor the average value of your_numeric_column for anomalies. If, for example, the average value suddenly increases or decreases beyond what is expected based on historical trends, the test will fail and alert you [column_anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/column-anomalies) [column_anomalies config](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/column-anomalies). --- If you want to add the average column anomaly test, you can use the example above and adjust it for your model and column. If you need more details or a specific example for your data, let me know!
s
@kapa.ai Winston users would like data quality checks on "Creation Date" Field in Item Branch Description Add and Elementary data anomaly checks to the parquet refined4 common refined4 ItemBranch • for rows that have a value can we check changes in the average date age? • the number of records that have dates over time • check if there are nulls, I think this is likely - name: item_branch__parquet_common_refined4 data_tests: - elementary.volume_anomalies: anomaly_direction: both ignore_small_changes: spike_failure_percent_threshold: 1 drop_failure_percent_threshold: 1 tags: ["elementary", "daily_elementary", "refined4", "common_refined4"] columns: - name: "%itembranchskey" quote: true data_tests: - unique: config: severity: error error_if: ">1" warn_if: ">1" - not_null add along with this the field is "itembranch.creation date" @kapa.ai how will the avergae works for a date column