https://linen.dev logo
Join Slack
Powered by
:wave: Hi everyone, I’m trying to define a configu...
# community-support
l
👋 Hi everyone, I’m trying to define a configuration for the freshness anomalies test at the project level so that I can reuse the same configuration across multiple models. I’ve attempted to place it in various sections of the 
dbt_project.yml
file (such as ‘elementary’, ‘tests’, etc.), but haven’t had any success. 😞 Is it possible to define this configuration at the project level? If so, could you guide me on the correct placement within the 
dbt_project.yml
file? Thank you in advance for your help!
r
They go in the "vars" section
l
Thank you. I’ve tried implementing this, but it doesn’t seem to work as expected. When I use the model-specific configuration, I receive a warning (there are some anomalous results). However, when I use the same configuration block at the project level, I don’t receive any warnings. Here’s my project-level configuration:
Copy code
vars: 
  timestamp_column: 'TIMESTAMP_COLUMN'
  anomaly_sensitivity: 1
  detection_period:
    period: day
    count: 2
  training_period:
    period: month
    count: 10
  time_bucket:
    period: week
    count: 1
  detection_delay:
    period: day
    count: 2
And here’s my model-specific configuration:
Copy code
models:
  - name: MODEL_NAME
    tests:
      - elementary.freshness_anomalies:
          timestamp_column: 'TIMESTAMP_COLUMN'
          anomaly_sensitivity: 1
          detection_period:
            period: day
            count: 2
          training_period:
            period: month
            count: 10
          time_bucket:
            period: week
            count: 1
          detection_delay:
            period: day
            count: 2
I can’t understand why the two configs are producing different results. Can anyone explain?
o
Hi @Lucie Nováková, setting the configuration in the 
vars
section of your dbt project will not work, as these are project variables and not test configurations. if you wish to set the configuration for elementary test on a scale higher then a specific test, you can do it on a model level, like so:
Copy code
models:
  - name: MODEL_NAME
    config:
      elementary:
        timestamp_column: 'TIMESTAMP_COLUMN'
        anomaly_sensitivity: 1
        detection_period:
          period: day
          count: 2
        training_period:
          period: month
          count: 10
        time_bucket:
          period: week
          count: 1
        detection_delay:
          period: day
          count: 2
    tests:
      - elementary.volume_anomalies
r
Documentation need to be updated, @Ofek Weiss https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period for most of these, the left hand pane example has a dbt_project.yml tab that shows setting these at a global level under "vars"
o
Oh my bad, @Renee Cooper i stand corrected, configuring it in the 
vars
section of the project should work drop shocked @Lucie Nováková can you share the graph of the different configurations? id like to try to see what is going on there
l
Thank you, @Ofek Weiss. I'm not sure about the specific graph you’re referring to. It’s possible that I do not have access to it as I’m currently working in a development environment, testing the config on sample data. However, I examined the queries that the test generates and I noticed a discrepancy in the time buckets. The 
days_back_start
is calculated differently: In the model-specific configuration, the date 2023-04-09 is used, which I believe corresponds to a 10-month training period. On the other hand, the vars configuration uses the date 2024-01-20. I’m not sure of the origin of this date.
o
@Lucie Nováková I believe I located the issue, ill do some testing to make sure and will update shortly 🙂
👍 1
Hey, So the issue is that 
detection_period
and 
training_period
specifically are only supported on a test level. This is a bug 🐛 , Im working on a fix 👷‍♂️, but in the meantime you will need to set those on a test level
r
Hi @Ofek Weiss, We’ve been having a similar issue with project level configs, model level configs and test level configs. For example, If I have the following configuration:
Copy code
- name: model_1
  meta:
     owner: "@me"
  description: '{{ doc("model_1_desc") }}'
  config:
     elementary:
        timestamp_column: timestamp_at_pt
  columns:
     - name: metric_1
     description: '{{ doc("metric_1_desc") }}'
     tests:
        - elementary.column_anomalies:
            column_anomalies:
              - sum
            where_expression: "some_field = 'some_value'"
            time_bucket:
              period: day
              count: 1
            tags: [ 'elementary' ]
The 
timestamp_column
does not propagate down to the 
column_anomalies
test. If I move the 
timestamp_column
under the test config, then the 
timestamp_column
config variable is passed into the test (see below):
Copy code
columns:
     - name: metric_1
     description: '{{ doc("metric_1_desc") }}'
     tests:
        - elementary.column_anomalies:
            timestamp_column: timestamp_at_pt
            column_anomalies:
              - sum
            where_expression: "some_field = 'some_value'"
            time_bucket:
              period: day
              count: 1
            tags: [ 'elementary' ]
Regardless though, I’ve been trying to set an 
anomaly_sensitivity
for this test, have tried a global config, model config and test config to no avail. When I have the 
anomaly_sensitivity
set at the test level I get an error saying that specific param is not available for 
column_anomalies
. The first thing I will say is the documentation for 
all_column_anomalies
and 
column_anomalies
is very confusing. But there does seem to be a bug with the 
column_anomalies
. I’ve tried discussing this with other elementary folks and no one has been able to give me an answer.
o
Hello @Robert Gillin, What version of 
elementary
are you using?
r
Copy code
version: 0.7.1
Actually 0.7.2
Confirmed this was resolved by upgrading to latest version. Thank you!
29 Views
Open in Slack
PreviousNext
Powered by