<@U088WCP3B4P> what does metric_values mean in dat...
# ask-elementary-ai
s
@kapa.ai what does metric_values mean in data_monitoring_metric of elementary ?
k
A message from kapa.ai
In Elementary, the
metric_value
column in the
data_monitoring_metrics
table represents the value of the specific metric being monitored for a given time bucket, dimension, or test run. The meaning of
metric_value
depends on the type of metric: - For a row count metric,
metric_value
is the number of rows in the table (or partition) for the specified time bucket. - For a freshness metric,
metric_value
is the difference in seconds between the last event in the bucket and the end of the bucket (e.g., end of the day). On BigQuery, this value may appear negative due to a known issue, but it still represents the time difference in seconds. You can use
ABS()
to get the positive value if needed. For example, a value of
-4
means the last event was 4 seconds before the end of the day [Slack discussion on freshness metric](https://slack.com/archives/C02CTC89LAX/p1658915396893639). - For other metrics (like null percent, min, max, etc.),
metric_value
will represent the calculated value for that metric in the context of the test and time bucket. Each row in
data_monitoring_metrics
corresponds to a metric collected for a specific time bucket (if a timestamp column is configured), dimension, and metric type. These values are used by Elementary to detect anomalies by comparing recent `metric_value`s to historical values [Elementary anomaly detection tests](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works). In summary,
metric_value
is the actual measured value of the metric Elementary is monitoring, and its meaning is determined by the metric type (e.g., row count, freshness, null percent, etc.) and the context of the test configuration.
s
@kapa.ai what does detection_period refer when my model has no time based column ?
k
A message from kapa.ai
When your model has no time-based column (i.e., you do not configure a
timestamp_column
), the
detection_period
in Elementary refers to the number of most recent test runs (not actual time periods) that are included in the detection set. Each test run calculates the metric for the entire table, and the detection period determines how many of these recent metric values are checked for anomalies. So, for non-timestamped tables: - The detection period is based on the number of test executions, not on calendar days or time buckets. - For example, if
detection_period.count
is set to 2, Elementary will check the last 2 test runs' metric values for anomalies. This is confirmed in the documentation and community discussions, which state that without a timestamp column, each test run produces a single metric for the whole table, and the detection and training periods are based on the number of runs, not actual time intervals [Anomaly Tests Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide) [Slack discussion on detection period](https://slack.com/archives/C02CTC89LAX/p1694016833712689). If you want the detection period to correspond to a specific number of test runs, set the
detection_period.count
accordingly.
s
@kapa.ai - name: efuse_table tests: - elementary.dimension_anomalies: dimensions: - efuse_id where_expression: "duplicate_efuse = true" config: severity: warn meta: owner: "@sn.kuttysivakumar" channel: swetha-elementary-test in this dimension_anomalies, I dont see a failed row count in the elementary_test_results table ! but in my efuse table I have 5K records with "duplicate_efuse = true". what could be missing in this test causing the failed row count to be null ?
@kapa.ai name: efuse_table tests: - elementary.dimension_anomalies: dimensions: - efuse_id where_expression: "duplicate_efuse = true" config: severity: warn meta: owner: "@sn.kuttysivakumar" channel: swetha-elementary-test in this dimension_anomalies, I dont see a failed row count in the elementary_test_results table ! but in my efuse table I have 5K records with "duplicate_efuse = true". what could be missing in this test causing the failed row count to be null ?
k
A message from kapa.ai
New message text here
s
@kapa.ai What is the relavant fiels for dimension_anomolies ?
k
A message from kapa.ai
For
dimension_anomalies
tests in Elementary, the most relevant fields are those that describe the detected anomaly and the context in which it occurred. According to the knowledge sources, the following fields are particularly important: - dimension: The column being monitored for distribution changes. - dimension_value: The specific value within the dimension that exhibited anomalous behavior. - metric_value: The observed value for the metric (e.g., count) for the given dimension value. - anomaly_score: The calculated score indicating how anomalous the metric value is compared to historical data. - anomaly_score_threshold: The threshold above which a value is considered anomalous. - training_avg: The average metric value over the training period. - training_stddev: The standard deviation of the metric value over the training period. - bucket_start and bucket_end: The time window (if a timestamp column is used) or test run for which the metric was calculated. - anomaly_description: A human-readable explanation of the anomaly, often constructed from the above fields. These fields are typically found in the anomaly scores tables (e.g.,
*_anomaly_scores
) and are used in the Elementary report to explain and visualize detected anomalies. The
failed_row_count
is not relevant for
dimension_anomalies
tests, as these tests are about distribution shifts rather than individual row failures [Slack: failed row count in anomaly tests](https://slack.com/archives/C02CTC89LAX/p1680365427971959) [Slack: anomaly scores fields](https://slack.com/archives/C02CTC89LAX/p1704689826965219). If you want to investigate anomalies, focus on these fields in the relevant tables and reports.
s
@kapa.ai what happens when I dont specify time_bucket for dimension_anomaly?
k
A message from kapa.ai
If you do not specify a
time_bucket
for a
dimension_anomalies
test in Elementary, the test will use the default time bucket, which is daily. This means that, if you have a
timestamp_column
configured, the distribution of the dimension will be monitored per day. If you do not have a
timestamp_column
, the test will simply count the total rows per dimension across the entire table for each test run, and compare those counts between runs [dimension_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies). In summary: - With a `timestamp_column`: Default time bucket is daily. - Without a `timestamp_column`: The test compares the total counts per dimension value for each test run. This behavior is described in the official documentation: > If
timestamp_column
is configured, the distribution is collected per
time_bucket
. If not, it counts the total rows per dimension. > [dimension_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies)
s
@kapa.ai how can i send alert to individuals on slack and not to a channel ?
k
A message from kapa.ai
To send alerts to individuals on Slack using Elementary, you should use the
owner
and
subscribers
fields in your model or test
meta
configuration. Specify the Slack usernames (with
@
and the Slack handle) in these fields. When an alert is triggered, Elementary will tag these users in the alert message, notifying them directly in the channel where the alert is posted. Example configuration:
Copy code
yaml meta:   owner: "@jessica.jones"   subscribers: ["@joe.joseph"]
This will tag
@jessica.jones
as the owner and
@joe.joseph
as a subscriber in the Slack alert. Note: Elementary does not support sending alerts as direct messages to users; alerts are always posted in a Slack channel, but you can tag individuals so they are notified. Other channel members will see the alert, but only the tagged users will be directly notified [Cloud alert properties in
.yml
files](https://docs.elementary-data.com/cloud/guides/alerts-configuration#alert-properties-in-yml-files) [OSS alert properties in
.yml
files](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alert-properties-in-yml-files) [Slack discussion on silent messages](https://slack.com/archives/C02CTC89LAX/p1688725512625389). If you want only specific individuals to be notified, consider muting the channel for everyone except those who are tagged, as suggested in the community discussions.
s
@kapa.ai my elementary report does not show dbt test
k
A message from kapa.ai
If your Elementary report does not show dbt test results, here are the most common causes and troubleshooting steps: 1. Check if
elementary_test_results
Table Has Data
- Make sure the table
elementary_test_results
exists in your data warehouse and contains data. If it does not exist, ensure you have run
dbt run --select elementary
to create the necessary tables. If it exists but is empty, make sure you have run
dbt test
after deploying the package [OSS Troubleshooting](https://docs.elementary-data.com/oss/general/troubleshooting). 2. Profile Configuration - The
elementary
profile in your
profiles.yml
must point to the same database and schema where the Elementary tables (like
elementary_test_results
) are created. If the report can see models but not test results, this is often a profile misconfiguration [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1657572412263959). 3. Consistent Targets - Ensure you are using the same
--target
for both
dbt run --select elementary
and
dbt test
. If you use different targets, the test results may be written to a different schema than the one the report is querying [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1683764076675589). 4. CLI and Package Version Compatibility - Make sure your Elementary CLI and dbt package versions are compatible. Incompatible versions can cause the report to not display test results [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1711981966108649). 5. Check for Disabled Results Upload - If you have set
disable_run_results: true
or
disable_tests_results: true
in your dbt config, Elementary will not upload test results, and the report will be empty [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1699310045488249). 6. Run Order - You must run
dbt run --select elementary
before running
dbt test
, so the Elementary models are created and ready to store test results [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1660748994945629). If you have checked all the above and still do not see test results, review the
edr.log
file for errors, and ensure your
profiles.yml
is in the correct location or specify the path with
--profiles-dir
if needed. If you need more help, please provide your
edr.log
and
profiles.yml
(with sensitive info redacted) for further troubleshooting.