https://linen.dev logo
Join Slack
Powered by
Hey folks, testing out Elementary as an anomaly de...
# community-support
m
Hey folks, testing out Elementary as an anomaly detection tool for our Data Warehouse, and am struggling a bit with 
column_anomalies
- tried many different settings, but can't get the tool to backfill and measure beyond 3 months. This works fine with volume_anomalies, but not with columns. Elementary can definitely read the data - last month, the report had September in it, but now it's gone and it only shows October. Searched the docs and the internets, but couldn't find anything related. Would appreciate a steer, feels like I'm missing something silly 😅 More information in thread 👇
DBT package 0.13.2 CLI package 0.13.2 General config in model
Copy code
config:
      elementary:
        timestamp_column: period_start
        time_bucket:
          period: month
          count: 1
        days_back: 186
        backfill_days: 186
Test setup
Copy code
tests:
          - elementary.column_anomalies:
              detection_period:
                period: month
                count: 1
              training_period:
                period: month
                count: 6
              column_anomalies:
                - sum
              config:
                severity: warn
for 
column_anomalies
, I only get three months, no matter what settings I supply:
for similarly set up 
volume_anomalies
this gives me a 5-month view (there is no more data):
n
Hey @Mateusz 👋🏼 Your configuration does look fine for me. Can you share the 
Configuration
section from the UI as well? Also in the UI, there is not a ‘where’ statement inside the 
Description
section, right? I am assuming these two tests are on the same model.
m
Hey @Noy Arie appreciate looking. Both test are on the same model. The config for the 
volume_anomalies
is similar:
Copy code
- elementary.volume_anomalies:
          detection_period:
            period: month
            count: 1
          training_period:
            period: month
            count: 6
          config:
            severity: warn
There is no 
where
in the description section (I've certainly haven't configured one). The config tab for the 
column_anomalies
and for 
volume_anomalies
is the same:
Copy code
Test Name:
column_anomalies
Timeframe:
1 month
Timestamp Column:
period_start
Anomaly Threshold:
3
is there a way to debug this? The UI gives me the query, but unfortunately the temp tables that drove it is deleted after the report is complete. Is there a switch that'd allow me to look into these tables maybe?
n
Can you share the 
test_params
field from 
dbt_tests
for these two tests?
m
for the `column_anomalies`:
Copy code
{
  "detection_period": {
    "period": "month",
    "count": 1
  },
  "training_period": {
    "period": "month",
    "count": 6
  },
  "column_anomalies": [
    "sum"
  ],
  "column_name": "total_bill_nzd",
  "model": "{{ get_where_subquery(ref('table_x', version='1')) }}"
}
for the `volume_anomalies`:
Copy code
{
  "detection_period": {
    "period": "month",
    "count": 1
  },
  "training_period": {
    "period": "month",
    "count": 6
  },
  "model": "{{ get_where_subquery(ref('table_x', version='1')) }}"
}
o
Hi @Mateusz, i suspect that there might be an issue since you have 
days_back
and 
backflll_days
set in your general model config, these deprecated old fields might conflict with the new 
detection_period
and 
training_period
params in the test config. can you try removing them/changing them to the new params and letting me know if it causes any improvements 🙏?
33 Views
Open in Slack
PreviousNext
Powered by