Hi I m trying to utilize dimensions to test column anomalies Elementary Community #community-support

Hi, I'm trying to utilize dimensions to test colum...

Krzysztof D

01/15/2025, 10:39 AM

Hi, I'm trying to utilize dimensions to test column anomalies separately for dimension column value. Correct me if I'm using this incorrectly. I have a table that contains some statistics that I would like to watch grouped by market_provider column. Since it's already aggregated I didn't care about the aggregation in the tests, I just used the first one on the list (max). The problem is that for yesterday for all the dimensions the value is 0 which should result in test fail since all prev days have values. I defined the tests as:

Copy code

models:
  - name: fin_event_stats
    config:
      elementary:
        timestamp_column: event_funded
    columns:
      - name: total_count
        tests:
          - elementary.column_anomalies:
              column_anomalies:
                - max
              dimensions:
                - market_provider
      - name: sales_count
        tests:
          - elementary.column_anomalies:
              column_anomalies:
                - max
              dimensions:
                - market_provider
      - name: request_count
        tests:
          - elementary.column_anomalies:
              column_anomalies:
                - max
              dimensions:
                - market_provider
      - name: status_found_count
        tests:
          - elementary.column_anomalies:
              column_anomalies:
                - max
              dimensions:
                - market_provider

Krzysztof D

01/15/2025, 10:40 AM

Here is the csv with the values:

Copy code

"MARKET_PROVIDER","EVENT_FUNDED","TOTAL_COUNT","SALES_COUNT","REQUEST_COUNT","STATUS_FOUND_COUNT"
a,"2024-12-31",19978,11496,317,301
a,"2025-01-01",8133,4058,171,171
a,"2025-01-02",16965,12404,579,564
a,"2025-01-03",19951,13843,506,490
a,"2025-01-04",16357,11313,241,231
a,"2025-01-05",13753,8801,163,154
a,"2025-01-06",20957,15136,558,550
a,"2025-01-07",24712,16246,179,177
a,"2025-01-08",17444,7611,175,168
a,"2025-01-09",15795,9345,212,203
a,"2025-01-10",20217,14112,342,320
a,"2025-01-11",15540,9572,90,83
a,"2025-01-12",18520,12107,105,93
a,"2025-01-13",21399,14785,549,496
a,"2025-01-14",8073,5753,0,0
b,"2024-12-31",45,4,0,0
b,"2025-01-01",20,7,0,0
b,"2025-01-02",23,8,0,0
b,"2025-01-03",83,10,0,0
b,"2025-01-04",214,196,0,0
b,"2025-01-05",6,1,0,0
b,"2025-01-06",14,2,0,0
b,"2025-01-07",15,10,0,0
b,"2025-01-08",51,12,0,0
b,"2025-01-09",63,34,0,0
b,"2025-01-10",19,10,0,0
b,"2025-01-11",63,39,0,0
b,"2025-01-12",9,7,0,0
b,"2025-01-13",59,56,0,0
b,"2025-01-14",5,5,0,0
c,"2025-01-10",38,2,2,2
c,"2025-01-11",38,4,3,3
c,"2025-01-12",46,4,2,1
c,"2025-01-13",102,23,3,2
c,"2025-01-14",36,11,0,0
d,"2024-12-31",5472,1430,206,200
d,"2025-01-01",4236,1063,110,110
d,"2025-01-02",8736,2244,300,298
d,"2025-01-03",7838,2160,301,300
d,"2025-01-04",2303,262,45,44
d,"2025-01-05",6773,2125,222,222
d,"2025-01-06",9928,2695,321,320
d,"2025-01-07",16391,2590,379,378
d,"2025-01-08",8171,2295,334,326
d,"2025-01-09",6376,1744,272,264
d,"2025-01-10",6285,1648,240,221
d,"2025-01-11",1491,180,28,28
d,"2025-01-12",5370,1468,158,158
d,"2025-01-13",9599,2643,305,290
d,"2025-01-14",4025,1011,0,0

Noya Offer

01/15/2025, 1:04 PM

Hey Krzysztof! You are using this correctly. This is probably a matter of the anomaly configuration (sensitivity, training and detection period, time buckets...) 🙂 Can you please send a screenshot of the test result in your edr report?

Krzysztof D

01/15/2025, 1:48 PM

Thanks for the replay. Sure, here it is:

Krzysztof D

01/15/2025, 2:04 PM

Are the thresholds calculated separately for each dimension value?

Noya Offer

01/16/2025, 9:05 AM

Yes, they are 🙂 With the Elementary default configurations and the values in the CSV, the 0 values aren't considered an anomaly. For example, the

request_count

values for

market_provider a

are quite varied, which creates a high standard deviation. The 0 values would only be considered an anomaly when the

anomaly_sensitivity

configuration will be set to 1.5 or under (the default sensitivity is 3). If you're worried about the 0 values, you can configure the

fail_on_zero

test monitor. Let me know if this helps!

Krzysztof D

01/16/2025, 11:33 AM

Thank you, yes it does help. Although I think the chart of the values and range limit in the report are a must! It's hard find out anything without the charts. 🙂

🙌 1

Noya Offer

01/16/2025, 11:59 AM

Thank you for your input! Noted 💪

keanu thanks 1

37 Views

Open in Slack

Previous Next