To implement a column anomaly test that fails if a daily match rate between two sources is below 90%, the best approach with Elementary is to create an intermediate model that calculates the daily match rate, and then apply an anomaly detection test to that metric. Elementary does not natively support cross-table comparisons or custom metrics directly in its anomaly tests. However, as suggested in the community and documentation, you can: 1. Create a derived model: Build a dbt model that, for each day, calculates the match rate between the two sources. For example, your model could output columns like

date

and

match_rate

. 2. Apply a column anomaly test: Use the

elementary.column_anomalies

test on the

match_rate

column of your derived model. You can configure the test to monitor for drops (using the

min

monitor) and set a

where_expression

to fail if the value drops below 0.9 (90%). Here’s a high-level example: Step 1: Create the derived model (SQL)

sql -- models/daily_match_rate.sql select   date_column as date,   count_matched / nullif(count_total, 0) as match_rate from (   -- Your logic to calculate count_matched and count_total per day ) sub

Step 2: Add the Elementary test in your dbt model

yaml models:   - name: daily_match_rate     columns:       - name: match_rate         tests:           - elementary.column_anomalies:               column_anomalies: [min]               timestamp_column: date               where_expression: "match_rate < 0.9"               time_bucket:                 period: day                 count: 1

This setup will monitor the

match_rate

daily and can be configured to alert if it drops below 90% on any given day. You can further tune the anomaly detection parameters as needed. This approach is also recommended by Elementary support for similar use cases where you need to monitor a custom metric derived from multiple tables or sources [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1677193288425909). If you need to strictly fail the test when the match rate is below 90% (not just detect anomalies), you might also consider a custom dbt test for that specific threshold, but for anomaly detection and trend monitoring, the above method is the best fit with Elementary.

Certainly! Here’s a breakdown of what the recommended approach is doing, based on the knowledge sources: 1. Create a Derived Model You build a dbt model (a SQL query saved as a model in your dbt project) that calculates the daily match rate between two sources. This model outputs, for each day, a value representing the percentage of matches (e.g., how many records in source A have a corresponding record in source B, divided by the total records). 2. Apply an Elementary Column Anomaly Test Once you have this model, you use Elementary’s

column_anomalies

test on the

match_rate

column. This test monitors the metric over time (daily, in this case) and can be configured to alert if the value drops below a certain threshold (like 90%). 3. Configuration Example The configuration in your

schema.yml

or

properties.yml

might look like this:

yaml    models:      - name: daily_match_rate        columns:          - name: match_rate            tests:              - elementary.column_anomalies:                  column_anomalies: [min]                  timestamp_column: date                  where_expression: "match_rate < 0.9"                  time_bucket:                    period: day                    count: 1

-

column_anomalies: [min]

tells Elementary to monitor for unusually low values. -

timestamp_column: date

means the test will look at the metric per day. -

where_expression: "match_rate < 0.9"

can be used to focus on days where the match rate is below 90%. -

time_bucket

settings ensure the test runs daily. Why do it this way? Elementary does not natively compare columns across tables. By creating a model that calculates the metric you care about (the match rate), you can then use Elementary’s anomaly detection on that metric, just like you would for any other column. This approach is recommended in the community for custom metrics that span multiple sources or require custom logic [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1677193288425909). If you need more details on how the anomaly detection works, you can refer to the [official documentation](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works). If anything is still unclear, please specify which part you’d like more detail on!

If the

match_rate

is 0.95 (i.e., 95%), the Elementary

column_anomalies

test as described will **pass**—it will not fail or alert—because 0.95 is above the 0.9 (90%) threshold specified in the

where_expression

. However, it's important to clarify that the

where_expression

in the test configuration is used to filter the data that the anomaly detection operates on, not to set a hard threshold for failure. Elementary's anomaly detection tests are designed to detect significant deviations from historical patterns, not to enforce strict thresholds. If the match rate of 0.95 is within the expected historical range, the test will pass. If it is an outlier compared to the historical data, it may still trigger an anomaly, depending on the sensitivity and configuration of the test. If you want to strictly fail the test whenever the match rate drops below 0.9, you would need to use a custom dbt test for that specific threshold, as Elementary's anomaly detection is not designed for hard threshold checks but for detecting anomalies based on historical trends and statistical analysis [How anomaly detection works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works). In summary: - If

match_rate

is 0.95, the test will pass unless 0.95 is considered an anomaly based on historical data and the test's sensitivity settings. - The test does not fail solely because the value is below 0.9 unless you implement a custom test for that purpose.

To achieve a WARN if there is a day within your 14-day detection period where the match rate is below 95%, you should use a custom dbt test for a strict threshold, because Elementary’s anomaly detection tests are designed to detect statistical anomalies, not to enforce hard thresholds. However, you can use the

severity: warn

config in dbt to make the test warn instead of error if it fails. Here’s how you can set this up: 1. Create a model that calculates the daily match rate (as described previously). 2. Add a custom dbt test to check for any day in the last 14 days where

match_rate < 0.95

, and set its severity to `warn`:

yaml # models/daily_match_rate.yml models:   - name: daily_match_rate     columns:       - name: match_rate         tests:           - dbt_utils.expression_is_true:               expression: "match_rate >= 0.95"               config:                 severity: warn                 where: "date >= current_date - interval '14 day'"

- This test will WARN if there is any day in the last 14 days where the match rate is below 95%. - The

severity: warn

config ensures it does not fail your pipeline, but issues a warning instead. Note: Elementary’s anomaly detection tests do not natively support hard threshold checks like this; they are for detecting statistical anomalies based on historical data. For strict threshold checks, a custom dbt test as shown above is the recommended approach, as also discussed in the knowledge sources [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1693358780626649). If you want to use Elementary’s anomaly detection for trend-based warnings, you can set the

anomaly_sensitivity

and

severity: warn

, but it will not strictly warn on every value below 0.95—only on values that are anomalous compared to history. Let me know if you need an example for the Elementary anomaly test configuration as well!

Use dbt tests when you want to validate explicit, fixed expectations about your data—such as checking for nulls, uniqueness, accepted values, or relationships. These tests are assertions: they pass or fail based on whether the data meets the defined criteria. They are ideal for business rules or data quality checks you know should always be true (e.g., "no nulls in primary key," "values must be in a specific set") [Detection and coverage](https://docs.elementary-data.com/best-practices/detection-and-coverage). Use Elementary anomaly detection tests when you want to monitor data quality metrics over time and detect unexpected changes or trends—such as sudden drops in row count, spikes in null rates, or shifts in value distributions. These tests are stateful and compare recent data to historical patterns, alerting you to anomalies that may indicate data issues you didn’t anticipate. They are especially useful for catching issues that are hard to define with static rules, such as gradual data drift or unexpected spikes [How anomaly detection works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works) [dbt, Packages and Elementary Tests](https://docs.elementary-data.com/features/data-tests/dbt-tests). In summary: - dbt tests: Use for fixed, known rules and assertions. - Elementary tests: Use for monitoring trends, detecting anomalies, and catching unknown or unexpected data issues. Both can be used together for comprehensive data quality coverage, and all results are visible in the Elementary UI [Data Tests Overview](https://docs.elementary-data.com/features/data-tests/data-tests-overview#test-configuration).

Yes, you can add Elementary tests to your source tables. Elementary anomaly detection and data quality tests can be configured on both models and sources in your dbt project. You simply define the tests in the YAML configuration for your sources, just as you would for models. This includes tests like freshness anomalies, volume anomalies, and schema changes, among others. This approach is confirmed in both the official documentation and community discussions [Add anomaly detection tests](https://docs.elementary-data.com/data-tests/add-elementary-tests) [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1676630491579719).

Certainly! Here is an example of how to add Elementary tests to your source tables in YAML, based on the knowledge sources:

yaml version: 2  sources:   - name: example_name     database: our_database_name     schema: our_schema_name     tables:       - name: table_name         meta:           elementary:             timestamp_column: "field"         tests:           - elementary.freshness_anomalies           - elementary.volume_anomalies           - elementary.schema_changes

In this example: - The

meta.elementary.timestamp_column

is used to specify the timestamp column for anomaly detection. - The

tests

section under each table lists the Elementary tests you want to run (e.g.,

freshness_anomalies

,

volume_anomalies

,

schema_changes

). This structure is confirmed in the knowledge sources and is the recommended way to configure Elementary tests on sources [Slack example](https://slack.com/archives/C02CTC89LAX/p1688995519653729).

The behavior you're seeing—where trendline dots represent intervals like "8/15/2025 8:00PM - 8/16/2025 8:00PM" instead of just "8/15"—is due to how Elementary defines time buckets based on your timestamp values. The time bucket boundaries are determined by the actual values in your

timestamp

field, including the time component, not just the date. This means that if your data's earliest timestamp is at 8:00PM, your daily buckets will start and end at 8:00PM, not midnight. Currently, Elementary does not provide a built-in configuration to force daily buckets to align strictly with calendar days (midnight to midnight). The bucket start and end times are based on the minimum timestamp in your data, as seen in the generated SQL and confirmed in community discussions. This can result in "intraday" buckets if your data does not have timestamps aligned to midnight [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1682055726978159). If you want your buckets to align with calendar days, you would need to ensure that your

timestamp

column is truncated to the date (e.g., using

DATE(timestamp)

or setting all times to midnight) in your model or source data before running the Elementary test. This way, the buckets will naturally align with calendar days. To summarize: - The bucket boundaries are determined by the actual timestamp values in your data. - To get buckets that align with calendar days, preprocess your data so the timestamp column only contains dates (midnight times). If you need help with the SQL to do this, let me know!

Your YAML configuration is mostly correct in terms of how you are adding Elementary tests to your source tables, but there are a few points and clarifications based on the knowledge sources: ### 1. Placement of

elementary

Configurations - The

elementary

configuration (such as

timestamp_column

) should be placed under the

meta

key for tables, not under

config

at the source level. The correct pattern is to use

meta.elementary.timestamp_column

at the table level, as shown in the [Slack example](https://slack.com/archives/C02CTC89LAX/p1688995519653729). ### 2. Test Definitions - The way you define Elementary tests under the

tests

key for each table is correct and matches the documented approach for adding anomaly detection tests to sources [Add anomaly detection tests](https://docs.elementary-data.com/data-tests/add-elementary-tests). ### 3. Use of

data_tests

- The use of

data_tests

for column-level tests (like

not_null

) is not standard dbt or Elementary syntax. The correct key is

tests

for both table-level and column-level tests, as shown in the knowledge sources. ### 4.

loaded_at_field

and

timestamp_column

-

loaded_at_field

is a dbt source freshness property, not an Elementary property. For Elementary, you should use

meta.elementary.timestamp_column

to specify the timestamp column for anomaly detection tests. ### 5. Example Correction Here is a corrected and simplified example for your

google_analytics

source:

yaml version: 2  sources:   - name: google_analytics     schema: ga4_exported_data     database: casper     loader: fivetran     loaded_at_field: "timestamp"     freshness:       warn_after: {count: 1440, period: minute}     tables:       - name: event         meta:           elementary:             timestamp_column: "date"         tests:           - elementary.freshness_anomalies:               tags: ["elementary"]           - elementary.volume_anomalies:               where_expression: "is_intraday = False"               time_bucket:                 period: day                 count: 1               training_period:                 period: day                 count: 14               detection_delay:                 period: day                 count: 1               detection_period:                 period: day                 count: 1               tags: ["elementary"]               config:                 severity: warn           - elementary.schema_changes:               tags: ["elementary"]               config:                 severity: warn       - name: item         loaded_at_field: event_timestamp

### 6. For the

heap

source - Use

tests

instead of

data_tests

for column-level tests. - If you want to add Elementary tests, follow the same pattern as above. ### Summary - Use

meta.elementary.timestamp_column

at the table level for Elementary. - Use

tests

for both table and column-level tests. -

loaded_at_field

and

freshness

are dbt source properties, not Elementary-specific. If you need more detailed corrections for the

heap

section or further clarification, let me know!

Based on the knowledge sources, here’s what can be determined about your issues: ### 1. Detection Delay Showing as 11 Days Instead of 1 There is no explicit documentation or known issue in the provided sources that explains why

detection_delay

would be interpreted as 11 days when you have configured it as 1 day. The documentation and examples consistently show that

detection_delay

should respect the value you set (e.g.,

count: 1

for one day) [Detection delay docs](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-delay). If you are seeing a different value in the output, this could be a bug or a misinterpretation in the UI or logs, but the knowledge sources do not provide a direct answer or workaround for this specific behavior. ### 2. Intraday Buckets Despite Using a Date Column If you are still seeing buckets like "8:00PM - 8:00PM" even after switching to a date column, this suggests that the underlying data or the way Elementary is interpreting the column may still include a time component, or the bucket alignment is based on the minimum value in your data, not strictly on calendar days. The knowledge sources confirm that Elementary determines bucket boundaries based on the actual values in your timestamp column, and if the earliest value is not midnight, buckets will not align to calendar days [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1682055726978159). Key points from the sources: - "The bucket boundaries are determined by the actual timestamp values in your data." - "To get buckets that align with calendar days, preprocess your data so the timestamp column only contains dates (midnight times)." If your column is truly of type

DATE

(not

TIMESTAMP

), and you are still seeing this issue, it may be worth double-checking the data type in your warehouse and ensuring that no time component is present. The knowledge sources do not provide further troubleshooting steps for this scenario. --- Summary: - The knowledge sources do not explain why

detection_delay

would be 11 days when set to 1. - Intraday bucket issues are likely due to how Elementary determines bucket boundaries, and you should ensure your date column is truly a date with no time component. - If these issues persist, they may be bugs or edge cases not covered in the documentation, and reaching out to Elementary support or opening a GitHub issue may be necessary. If you need help with SQL to ensure your column is a true date, let me know!

Certainly! Here’s content for your slide, based strictly on the knowledge sources: --- ### What is Elementary? (High-Level Introduction) Elementary is a dbt-native data observability platform designed for data and analytics engineers. It integrates directly into dbt workflows, enabling teams to ensure the reliability and quality of their data products. Elementary continuously monitors data pipelines, detects anomalies and performance issues, and provides real-time incident management. The platform leverages existing dbt tests and configurations, offering a seamless, code-first approach to monitoring data health. It supports both open-source and cloud-based solutions, with the cloud platform providing advanced features like automated freshness and volume monitoring, ML-powered anomaly detection, a simple data catalog, and rich integrations—all without accessing your raw data [Elementary Introduction](https://docs.elementary-data.com/introduction) [Cloud Platform](https://docs.elementary-data.com/cloud/introduction). --- ### Elementary Tests vs. dbt Tests - dbt Tests: - Used to validate explicit, fixed expectations about your data (e.g., no nulls, uniqueness, accepted values, relationships). - These are assertions: they pass or fail based on whether the data meets the defined criteria. - Ideal for business rules or data quality checks you know should always be true. - Examples:

not_null

,

unique

,

accepted_values

, and tests from packages like

dbt-utils

and

dbt-expectations

 [Detection and coverage](https://docs.elementary-data.com/best-practices/detection-and-coverage). - Elementary Tests: - Focused on monitoring data quality metrics over time and detecting unexpected changes or trends (anomalies). - These tests are stateful and compare recent data to historical patterns, alerting you to anomalies that may indicate unknown or unexpected data issues. - Examples: sudden drops in row count, spikes in null rates, schema changes, or shifts in value distributions. - Especially useful for catching issues that are hard to define with static rules, such as data drift or unexpected spikes [How anomaly detection works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works) [dbt, Packages and Elementary Tests](https://docs.elementary-data.com/features/data-tests/dbt-tests). In summary: - Use dbt tests for fixed, known rules and assertions. - Use Elementary tests for monitoring trends, detecting anomalies, and catching unknown or unexpected data issues. - Both can be used together for comprehensive data quality coverage, and all results are visible in the Elementary UI [Data Tests Overview](https://docs.elementary-data.com/features/data-tests/data-tests-overview#test-configuration).

Certainly! Here’s content for your slide, strictly based on the knowledge sources and focused on the open-source (OSS) version of Elementary: --- ### What is Elementary? (High-Level Introduction) Elementary OSS is a dbt-native data observability tool that works alongside the Elementary dbt package. It empowers data and analytics engineers to ensure the reliability and quality of their data products by continuously monitoring dbt pipelines for anomalies and data quality issues. Elementary OSS leverages your existing dbt tests and configurations, collects metadata and test results in your data warehouse, and provides a comprehensive, self-hosted observability report. It is designed to be code-first, integrates seamlessly with dbt workflows, and does not require access to your raw data—only to logs, test results, and metadata stored in a dedicated schema [Elementary OSS](https://docs.elementary-data.com/oss/oss-introduction) [Introduction](https://docs.elementary-data.com/introduction). --- ### Differences Between Elementary Tests and dbt Tests dbt Tests: - Validate explicit, fixed expectations about your data (e.g., no nulls, uniqueness, accepted values, relationships). - Are assertions that pass or fail based on whether the data meets the defined criteria. - Ideal for business rules or data quality checks you know should always be true. - Examples include built-in dbt tests and tests from packages like

dbt-utils

and

dbt-expectations

 [Detection and coverage](https://docs.elementary-data.com/best-practices/detection-and-coverage). Elementary Tests: - Focus on monitoring data quality metrics over time and detecting unexpected changes or trends (anomalies). - Compare recent data to historical patterns, alerting you to anomalies that may indicate unknown or unexpected data issues. - Useful for catching issues that are hard to define with static rules, such as data drift or sudden spikes in null rates or row counts. - Configured and executed like native dbt tests, and results are included in the Elementary observability report [How anomaly detection works](https://docs.elementary-data.com/data-tests/how-anomaly-detection-works) [Data Tests Overview](https://docs.elementary-data.com/features/data-tests/data-tests-overview). Summary: - Use dbt tests for fixed, known rules and assertions. - Use Elementary tests for monitoring trends, detecting anomalies, and catching unknown or unexpected data issues. - Both can be used together for comprehensive data quality coverage, with all results visible in the Elementary OSS report.

I'm sorry, but the knowledge sources provided do not explicitly define what "OSS" stands for. However, in the context of the documentation, "OSS" refers to the open-source version of Elementary, as seen in phrases like "Elementary OSS is a CLI tool" and "Elementary open-source is free to use" [Elementary OSS](https://docs.elementary-data.com/oss/oss-introduction) [Github Issue](https://github.com/elementary-data/elementary/issues/1625). If you need an exact definition, please refer to official documentation or reach out to the Elementary team.

A common use case for the

dimension_anomalies

test is monitoring the distribution of event types in an events table. For example, if you have an events table with a column

event_name

, you can use the

dimension_anomalies

test to detect unexpected drops or spikes in the number of rows for each event type (such as "registration", "purchase", etc.). This helps you catch issues like a sudden drop in "registration" events from a specific country, which could indicate a data pipeline or upstream issue. The test is best configured on low-cardinality fields and can also be used with combinations of dimensions (e.g.,

event_name

and

country_name

) to monitor more granular segments [Slack example](https://slack.com/archives/C02CTC89LAX/p1682676629448539) [dimension_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies).

Certainly! Here are concrete use cases for both the

dimension_anomalies

and

event_freshness_anomalies

tests, based strictly on the knowledge sources: --- ### Dimension Anomaly Test Use Case: Suppose you have an events table that logs different types of user actions, such as

event_name

(e.g., "registration", "purchase", "login") and

country_name

. You want to detect if there is a sudden drop or spike in the number of "registration" events from the US, which could indicate a data pipeline issue or a problem with user onboarding in that region. By configuring the

dimension_anomalies

test on the

event_name

and

country_name

columns, Elementary will monitor the row count for each combination over time and alert you to unexpected changes in the distribution. This is especially useful for catching issues in low-cardinality fields or specific business segments [dimension_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies) [Slack example](https://slack.com/archives/C02CTC89LAX/p1682676629448539). --- ### Event Freshness Anomaly Test Use Case: Imagine you have a streaming events table where each row has an

event_timestamp

(when the event actually occurred) and an

update_timestamp

(when the event was loaded into your data warehouse). You want to ensure that events are being ingested promptly and detect if there are delays in data loading. By using the

event_freshness_anomalies

test, you can monitor the time difference between when an event occurs and when it is loaded. If the delay suddenly increases (for example, due to a lag in your ingestion pipeline), the test will alert you to this anomaly. This is particularly valuable for continuous or streaming data sources where timely data availability is critical [event_freshness_anomalies documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/event-freshness-anomalies). ---

The

--timezone

(or

-tz

) flag for the Elementary CLI is used to specify the timezone that will be applied when calculating the time fields in alerts and reports. This allows you to control how timestamps are displayed and interpreted, rather than relying on the default (which is typically UTC or the local machine/browser timezone). You can use the flag like this in your CLI command:

edr report --timezone "Europe/Paris"

or

edr monitor --timezone "America/New_York"

A list of valid timezones can be found [here](https://en.wikipedia.org/wiki/List_of_tz_database_time_zones) under the "TZ database name" column. This flag ensures that the time fields in your reports and alerts are shown in the timezone you specify, which is useful for teams working in specific regions or needing consistency across environments [CLI options documentation](https://docs.elementary-data.com/oss/cli-commands#cli-options). If you want to set the timezone via an environment variable, you can use:

export EDR_REPORT_TIMEZONE="Europe/Paris"

and then run your CLI command as usual. This flag is available in the latest versions of the Elementary CLI [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671131472596299).

The error message

Error: No such option: --timezone

indicates that the version of Elementary CLI you are using (0.19.1) does not support the

--timezone

flag. According to the knowledge sources, the

--timezone

flag is available in the latest versions of the Elementary CLI, and there have been cases where users needed to upgrade to access new CLI options [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671131472596299). To resolve this, you should upgrade your Elementary CLI to the latest version as suggested in the output:

/opt/anaconda3/envs/elsa_env/bin/python3.12 -m pip install --upgrade elementary-data

After upgrading, the

--timezone

flag should be available for use as described in the documentation [CLI options documentation](https://docs.elementary-data.com/oss/cli-commands#cli-options). If you continue to experience issues after upgrading, please check the help output with

edr report --help

to confirm the available options in your installed version.

Probably a UI bug. ?