Is there an anomaly test for % of unique values? I...
# ask-elementary-ai
a
Is there an anomaly test for % of unique values? I.e. if column my_id typically has no duplicated values I want to be alerted if there is a higher (or lower) than expected number of duplicates.
k
A message from kapa.ai
Elementary does not provide a built-in anomaly test specifically for the percentage of unique values in a column (i.e., monitoring the proportion of unique values and alerting on unexpected drops or spikes in uniqueness) as a native configuration option in its anomaly detection tests. However, you can achieve this by leveraging dbt-expectations' test
expect_column_proportion_of_unique_values_to_be_between
, which allows you to set thresholds for the proportion of unique values and configure warning and error severities. For example, you can set a warning if the proportion drops below 0.98 and an error if it drops below 0.9. This approach is discussed in the community and is compatible with Elementary's test coverage and reporting, as Elementary integrates results from dbt-expectations and other dbt tests into its UI and alerting system:
Copy code
yaml tests:   - dbt_expectations.expect_column_proportion_of_unique_values_to_be_between:       min_value: 0.91  # Required       max_value: 1.0   # Upper limit       config:         severity: warn         error_if: "<0.9"         warn_if: "<0.984"
This will alert you if the uniqueness of your column falls below the expected range, as you described. For more details, see the discussion in the community Slack and the reference to the dbt Test Hub for further explanations about available tests and their configurations [Slack discussion on uniqueness test](https://slack.com/archives/C02CTC89LAX/p1738907631163159). If you want to monitor the number of distinct values over time as an anomaly (e.g., sudden drop or spike in distinct values), a common workaround is to create a derived table that calculates the count of distinct values per time bucket, and then apply an Elementary anomaly detection test (such as
column_anomalies
with the
sum
monitor) on that derived metric. This approach is described by community members as a way to monitor distinct value counts using Elementary's anomaly detection capabilities [Slack discussion on distinct value anomalies](https://slack.com/archives/C02CTC89LAX/p1702564369913829). In summary: - For static thresholds on uniqueness, use dbt-expectations' test as shown above. - For anomaly detection on the count of unique values over time, use a derived metric and apply Elementary's anomaly detection tests to it. Let me know if you need an example of the derived metric approach!