https://linen.dev logo
Join Slack
Powered by
Hi all. I am trying to explore column anomaly dete...
# community-support
d
Hi all. I am trying to explore column anomaly detection the following is the test configuration - name: water_quality columns: - name: pH config: elementary: timestamp_column: current_timestamp tests: - elementary.column_anomalies: column_anomalies: average detection_period: period: day count: 1 training_period: period: day count: 5 time_bucket: period: day count: 1 I have created a dummy data for 30 days with 2 records a day. My understanding is that since the time bucket is per day. and training_period is 5 day. for every 6th day it will try to predict anomaly based on the previous 5 days data. but when I build the report i am getting the following message. "Not enough data to calculate anomaly score" Please advise if my understanding is wrong.
n
Hey Dharmin! Your understanding is correct, I believe the issue here is syntax and not about a lack of training data (that is configurable as you said). Please try making this small change to the test configuration:
Copy code
tests:
          - elementary.column_anomalies:
              column_anomalies:
                - average
Let me know if your test works properly after this change 🙂
d
HI @Noya Offer Thank you so much for your response i have changed the configuration as said but it still says the same 124645 1 of 8 START test elementary_source_column_anomalies_dq_source_water_quality_average__pH__day__1__day__1__day__3 [RUN] 124645 Elementary: Started running data monitors on: HIVE_METASTORE.DQ_SOURCE.WATER_QUALITY pH 124653 Elementary: Finished running data monitors on: HIVE_METASTORE.DQ_SOURCE.WATER_QUALITY pH 124654 Elementary: Not enough data to calculate anomaly scores on 
test.tiger_dq_anomaly_detection.elementary_source_column_anomalies_dq_source_water_quality_average__pH__day__1__day__1__day__3.253f892125
tests: - elementary.column_anomalies: column_anomalies: - average detection_period: period: day count: 1 training_period: period: day count: 3 time_bucket: period: day count: 1
n
Of course! Can you send your CSV so I can see the values you're testing?
d
aqua_poc.csv
aqua_poc.csv
Thank you for the prompt response. It really means so much.
🙏 1
Found out the issue, It was due to column datatype
😄
n
Happy to hear! Which column?
d
timestamp column
was string instead of timestamp
🙌 1
p
@Noya Offer @Dharmin Siddesh J hi guy, did you face a issue for use case with bucket monthly? I have a table with samples as images. I configured elementary test - name: cnt_cust tests: - elementary.column_anomalies: column_anomalies: - average timestamp_column: 'start_date_of_month' training_period: period: month count: 12 detection_period: period: month count: 2 time_bucket: period: month count: 1 tags: ['elementary'] anomaly_sensitivity: 2 Period month here is used to check value variance for each month which already was aggregated The results shall be expected to appear + training_start: 1 Jan 2024 + training_end: 1 Jan 2025 + start_time: 1 Jan 2025 + end_time: 1 Feb 2025 However, actually, the result showed in table test_result_rows + training_start: 1 Mar 2024 + training_end: 1 Feb 2024 + start_time: 1 Jan 2025 + end_time: 1 Feb 2025 Please help to advise for this case. Thanks.
n
Hey @Passat Truong! The issue is a wrong training period, correct? The test configurations look good. • Is start_date_of_month a date column? • Can you please share the results you're seeing in the report?
p
Hey @Noya Offer • Is start_date_of_month a date column? => Yes. Aggregrated metric by month and check its anomaly • Can you please share the results you’re seeing in the report? Checked in table test_result_rows. It showed 11 records corresponding with 11 latest months and it took 12 latest months including detected value to calculate statistics and compared detected value to this. Pls take a look at parameter: training_set_size, training_start, training_end in attached image. Plus, training period shall be at least 2 record, isn't it? If any concerns, I am ready to share more details.
Also, I wonder that if training period from 1/3/2024 to 1/2/2025 but data only show 1/1/2025, it means training end in 1/2/2025 automatically extract from time of running (in fact 2/2025). However, its calculated statistics is applied from 1/2/2024 to 1/1/2025. This is its results from the last record which corresponded to training_set_size at 12. _{"value": 138.0, "average": 131.0, "min_value": 108.63119054657498, "max_value": 153.36880945342503, "start_time": "2025-01-01T000000", "end_time": "2025-02-01T000000", "id": "debc705762c399aabd0420ae8b17b3ec", "metric_id": "beffc1939b9af196508bbbf979c4e98e", "test_execution_id": "f632fb67-5a11-4274-acda-f703994009ee.test.dwh.elementary_column_anomalies_anomaly_test_2__average__qtty__month__2__month__1__start_date_of_month__month__12.371fba35f0", "test_unique_id": "test.dwh.elementary_column_anomalies_anomaly_test_2__average__qtty__month__2__month__1__start_date_of_month__month__12.371fba35f0", "detected_at": "2025-02-17T034409.692382", "full_table_name": "DWH_UAT_RAW.REPORTING.ANOMALY_TEST", "column_name": "qtty", "metric_name": "average", "anomaly_score": 0.6258714854337667, "anomaly_score_threshold": 2, "anomalous_value": null, "bucket_start": "2025-01-01T000000", "bucket_end": "2025-02-01T000000", "bucket_seasonality": "no_seasonality", "metric_value": 138.0, "min_metric_value": 108.63119054657498, "max_metric_value": 153.36880945342503, "training_avg": 131.0, "training_stddev": 11.18440472671251, "training_set_size": 12, "training_start": "2024-03-01T000000", "training_end": "2025-02-01T000000", "dimension": null, "dimension_value": null, "anomaly_description": "In column qtty, the last average value is 138.000. The average for this metric is 131.000.", "max_bucket_end": "2025-02-01T000000", "is_anomalous": false}_ If possible, could you explain logic and how to work? @Noya Offer Thank you.
Hi @Noya Offer Attached image for calculated metrics at _"average": 131.0, "min_value": 108.63119054657498, "max_value": 153.36880945342503_
Hi @Noya Offer please help to advise your case. Thank you so much.
n
Hey Passat, can you please show a screenshot of your test results in the report?
p
Hey @Noya Offer, I already attached all file above. Pls look at them
@Noya Offer Please help to advise this case. Your feedback is extremely valuable at this time. Because we are stucking with it
🙌 1
n
Hey @Passat Truong, sent you a DM 🙂
36 Views
Open in Slack
PreviousNext
Powered by