Hi All, We have an

all_columns_anomalies

test running against an aggregated table, where each column represents an aggregation of event data. Due to the fact that each column represents a different aspect we want to aggregate, and each column may have different dates where we know peaks occur, we want to exclude these peaks to ensure the expected values aren't skewed. What would be the best practice to handle this scenario? Thanks!

Hey Noam, interesting question! 🙂 We'll need to understand a bit more to offer the best solution. Are the dates with the peaks consistent for all columns? Do they have a certain seasonality? Do you know the values of the peaks or are they changing?

Hi @Noya Offer, Thank you for your response. 1. The dates of the peaks are not consistent across all columns, and there is no seasonality. 2. The values at the peaks fluctuate, with no clear upper or lower limits. For context, we use the

all_columns_anomalies

, which currently runs against 70 columns and is expected to expand in the future. Attached below is a screenshot illustrating the issue. You can see how the peaks create a broader window of expected values. We are trying to think of an automation for removing or ignoring these peaks. Thanks!

Thank you. Every anomaly test has the option to set the parameter

anomaly_exclude_metrics

, where you can exclude either by dates, time buckets or values. This way, these metrics won't be included in the expected range calculation. If you know the peak dates, you can use this to ignore them. It's a difficult problem because the peak dates/peak values vary for each column. Do you have any way to group the columns into separate views so you don't need to set individual tests per column?

Hey, im attaching how we currently have it defined:

- name: features_usage_per_day
    description: "aggregate feature usage per day, each field represent the usage of the a specific feature Total fields is 70"
    tests:
      - elementary.all_columns_anomalies:
          meta:
            description: "Check for anomalies in all usage-numeric columns"
          exclude_regexp: "^(usage_date)$"
          training_period:
            period: day
            count: 168 
          anomaly_sensitivity: 4
          timestamp_column: usage_date
          seasonality: day_of_week
          detection_period:
            period: day
            count: 1
          time_bucket:
            period: day
            count: 1 
          column_anomalies:
              - average
          tags: ['daily_anomalies']

I saw the

anomaly_exclude_metrics

its pretty manual even if i split them up into view, i would maybe go back and alter the result instead of excluding them. 1. How would you recommend to approach this? 2. What is considered to be best practice regarding the orchestration of the elementary anomaly tests? The issue we are currently struggling with is that we trigger them in the same job, causing the job itself to be very long, is there a way to work paralel?

Hey, this usecase is pretty challenging 🙂 Did you have any thoughts on how to efficiently alter the results? Also, trying to understand which (real) anomalies do you expect to see in your data. Which of these metrics are you worried about? https://docs.elementary-data.com/data-tests/anomaly-detection-tests/all-columns-anomalies We can have a look if a flag like

exclude_anomalies_in_training

would make sense, or maybe add features to

anomaly_exclude_metrics

(ignore a certain range of standard deviation). If you are up for we are happy to discuss open source contribution. As for the second question, not sure which orchestrator you're using, but I would suggest the typical dbt best practices around triggering, leveraging things like tags or other parameters to use in your

—select

field. https://docs.getdbt.com/reference/commands/test