<@U088WCP3B4P> I am using the latest version of th...
# ask-elementary-ai
i
@kapa.ai I am using the latest version of the package, I want to add a freshness anomaly test with hour bucket, and hour training_period and detection period as follows
Copy code
data_tests:
      - elementary.freshness_anomalies:
          timestamp_column: load_timestamp
          training_period:
            period: hour
            count: 336
          time_bucket:
            period: hour
            count: 2
          detection_period:
            period: hour
            count: 2
I have seen this was previously a problem and was solved in one recent version, however I still get the error
Copy code
Missing mandatory configuration: ['backfill_days']
  
  > in macro validate_mandatory_configuration (macros/edr/tests/test_configuration/get_anomalies_test_configuration.sql)
  > called by macro get_anomalies_test_configuration (macros/edr/tests/test_configuration/get_anomalies_test_configuration.sql)
  > called by macro test_table_anomalies (macros/edr/tests/test_table_anomalies.sql)
  > called by macro test_freshness_anomalies (macros/edr/tests/test_freshness_anomalies.sql)
  > called by test elementary_freshness_anomalies_stg__cloudflare_logs_hour__2__hour__2__load_timestamp__hour__336 (models/cloudflare/_model.yml)
even if in the documentation it looks possible to use hour as value for everything
k
A message from kapa.ai
You are correct that the documentation for Elementary's anomaly detection tests (including freshness_anomalies) lists support for hour-level granularity in the configuration of time_bucket, training_period, and detection_period parameters. However, despite this, there are multiple reports and confirmations from both the Elementary team and users that configuring detection_period (and related parameters) with period: hour still results in the error:
Copy code
Missing mandatory configuration: ['backfill_days']
This is a known issue and has not yet been fully resolved, even in recent versions. The root cause is that, under the hood, the detection_period and training_period parameters are still being translated to "days_back" and "backfill_days", and the code currently requires these to be at least one day (i.e., 24 hours or more). As a result, using period: hour with a count less than 24 will trigger this error, even though the documentation suggests it should work. This limitation is confirmed in several sources: - A GitHub issue explicitly states that detection_period must be longer than one day, and that using a shorter period (e.g., 2 hours) will not work and will result in the error you are seeing. There is no current workaround to allow for true hourly detection periods, though there is interest from the community and the Elementary team in supporting this in the future. [Not able to set up detection period - 'hour' for column_anomalies tests](https://github.com/elementary-data/elementary/issues/1646) - Slack discussions and other GitHub issues confirm that this is a bug/limitation, and that the only current workaround is to set detection_period to at least one day. [Slack thread on hourly detection_period](https://slack.com/archives/C02CTC89LAX/p1704990385240799) - The documentation does not reflect this limitation, but the implementation does not yet support sub-day detection periods for these tests. Summary: Even though the documentation suggests you can use hour-level granularity for detection_period and training_period, in practice, you must set detection_period to at least one day (period: day, count: 1) to avoid the "Missing mandatory configuration: ['backfill_days']" error. Hourly detection periods are not yet supported, and there is no current workaround to enable this behavior in the latest version of the package. If you need more granular (hourly) anomaly detection, you will need to wait for a future release where this is properly supported, or consider alternative approaches as suggested in the GitHub issues and Slack threads referenced above.