Hi Elementary team :blob-wave: We're still really ...
# community-support
a
Hi Elementary team blob wave We're still really new to the tool however we're running into an issue with the volume anomaly test where rows are either being counted incorrectly or not at all. This is a snippet of our config for the table we're testing. We are using Snowflake as a our data warehouse.
Copy code
- name: mdWeighIn
    config:
      elementary:
        timestamp_column: "_AIRBYTE_EMITTED_AT"
    tests:
    - elementary.volume_anomalies:
        time_bucket:
          period: day
          count: 1
        detection_period:
          period: day
          count: 3
        training_period:
          period: day
          count: 15
        config:
          severity: warn
        tags: ['elementary']
The attached screenshot is what gets generated. You can see that the count is all 0 for the Training period. The detection period is also calculating incorrectly for all dates but surprisingly the most recent day (08/04) does record the correct row count. We double checked by running a quick query ourselves on this table and have confirmed that there are definitely rows generated for each day of
_AIRBYTE_EMITTED_AT
so we currently don't understand why the test results are giving little to no row counts. Any help on anything we may have missed would be really appreciated!
n
Hey @Alex! The configuration looks good, just making sure - did this configuration lead to a detection period of 4 days, as the UI screenshot shows?
a
Hey Noya, thanks for getting back to me! Yea it did - that was the other thing, not sure why there is a 4 day detection window when it was set to 3?
n
I believe it's all connected, let's start troubleshooting this. Can you try going through the first two steps here? I want to make sure if the data that's collected already has this error. Also, is there anything in the way this table is built that can cause this? Do you delete rows from it?
a
I'm looking into the steps but some of the columns from our
data_monitoring_metrics
and `metrics_anomaly_score`tables don't match the query given in troubleshooting steps. Screenshot of what columns we have in our diagnostic tables. e.g
metric_timestamp
doesn't exist in
data_monitoring_metrics
metric_value, zscore, severity, detected_at
doesn't exist in
metrics_anomaly_score
Also, is there anything in the way this table is built that can cause this? Do you delete rows from it?
The table is an incremental model and we do have delete+insert strategy in our config. Is that something that would affect the anomaly detection? This table is currently in my dev database and was simply cloned from prod, so i didn't build the model. All i ran was
dbt test --select tag:elementary
👀 1
n
You're completely right about the outdated queries! Thank you, it's all fixed now.
The volume test does a simple
count(*)
, I believe there's no time where the table is completely empty (after deleting), right? Checking the metrics would be very helpful for this