<@U088WCP3B4P>, regarding automated volume tests i...
# ask-elementary-ai
d
@kapa.ai, regarding automated volume tests in Elementary Cloud, I am seeing that if the table already has a big number of records, it may not mark as an error or warning if we stop receiving new records for a while, as it will take many days to reach the lower threshold. How can I deal with this? I see that the manual volume test can be configured for Daily buckets, what about automated ones?
k
A message from kapa.ai
You're correct that with automated volume anomaly detection in Elementary, if a table already has a large number of records, a sudden stop in new records may not immediately trigger an alert. This is because the anomaly detection is based on deviations from historical row counts, and a large, stable table will require a significant drop (relative to its historical average and standard deviation) before an anomaly is flagged. This can result in a delay before a "no new data" situation is detected as anomalous, especially if the table's row count remains high despite no new records being added. How to address this: - Time Buckets: You can configure the volume anomaly test to use daily buckets by setting the
time_bucket
parameter. This is supported in both manual and automated tests. For example, if your data is expected daily, you can set:
Copy code
yaml   time_bucket:     period: day     count: 1
This will make the test evaluate row counts per day, making it more sensitive to days with no new data, even if the total row count remains high. This approach is especially useful for tables that are updated incrementally or on a regular schedule. If your data arrives weekly, you can adjust the bucket accordingly (e.g.,
period: day
,
count: 7
) [see example and discussion here](https://slack.com/archives/C02CTC89LAX/p1678140907623319). - No Timestamp Mode: If you do not specify a
timestamp_column
, Elementary will compare the row count at each test run to previous runs. However, this approach is less sensitive to missing data on a specific day if the total row count is still high. Using time buckets with a timestamp column is generally more effective for detecting missed loads [see discussion](https://slack.com/archives/C02CTC89LAX/p1678140907623319). - Complement with Freshness/Recency Tests: For immediate detection of missing data, consider adding a dbt freshness or recency test (e.g.,
dbt_utils.freshness
or
dbt_expectations.expect_row_values_to_have_recent_data
). These tests are designed to alert you when no new data has arrived within an expected timeframe, regardless of the total row count. Elementary supports displaying these tests in its reports and can alert on them as well [see here](https://slack.com/archives/C02CTC89LAX/p1706042023272269). Summary: To make automated volume anomaly detection more responsive to missing daily data, configure daily time buckets and, if possible, use a timestamp column. For even faster or more deterministic alerts, supplement with freshness or recency tests, which are supported in Elementary's reporting and alerting. If you need more specific configuration details for "automated" (as opposed to manual) tests in Elementary Cloud, the knowledge sources do not provide explicit documentation on differences, but the general configuration principles above apply to both modes. If you have further questions about automated test configuration in Elementary Cloud, please provide more details or consult the official documentation or support channels.
d
@kapa.ai how can I set the time bucket in the automated cloud monitor for volume?
k
A message from kapa.ai
I'm sorry, but the knowledge sources do not contain explicit information about configuring the time bucket for automated volume monitoring in Elementary Cloud's automated cloud monitor. The documentation and Slack discussions detail how to set the
time_bucket
parameter for manual (code-based) tests in your dbt project, but they do not specify whether or how you can set or override the time bucket for automated monitors created directly in the Elementary Cloud UI or via automated processes. If you need to control the time bucket, the documented approach is to define it in your dbt test configuration using the
time_bucket
parameter, for example:
Copy code
yaml tests:   - elementary.volume_anomalies:       timestamp_column: 'loaded_at'       time_bucket:         period: day         count: 1
[See documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket) For automated cloud monitors, if there is a UI or API option to set the time bucket, it is not described in the provided knowledge sources. I recommend reaching out to Elementary support or consulting the latest official documentation for guidance on this specific feature.
d
@kapa.ai, and could I set a volume test with a time bucket of 3 hours? Meaning that every 3 hours I would measure the number of records and see if there are any deviations
k
A message from kapa.ai
Yes, you can set a volume anomaly test in Elementary with a time bucket of 3 hours. The
time_bucket
configuration supports the
hour
period, and you can set the
count
to 3 to create 3-hour buckets. Here’s how you would configure it in your dbt test:
Copy code
yaml tests:   - elementary.volume_anomalies:       timestamp_column: <your_timestamp_column>       time_bucket:         period: hour         count: 3
This configuration will split your data into 3-hour intervals and monitor the row count for each interval, allowing you to detect deviations every 3 hours. This is supported for anomaly detection tests with a
timestamp_column
configured. Please note that the default is daily buckets, so you need to explicitly set this if you want 3-hour granularity [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket). If you are using automated cloud monitors (not code-based tests), the knowledge sources do not specify whether this configuration is available directly in the UI or automated setup. For dbt-based tests, the above configuration is supported.