https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> How can I find the compiled code fo...
# ask-elementary-ai
f
@kapa.ai How can I find the compiled code for an elementary test ? I need to debug it.
k
A message from kapa.ai
You can find the compiled SQL code for an elementary test in several places, similar to standard dbt tests: 1. dbt_run_results table: Look for the 
compiled_sql
column. 2. elementary_test_results table: Check the 
test_results_query
column. 3. Elementary report: The compiled SQL is shown in the test's details section, along with sample data. 4. Alerts: If you have alerts configured, the compiled SQL can also be included in the alert details. This allows you to review and debug the exact SQL that was generated and executed for your test. If you want to see the compiled code even when a test passes, you can access it in the report or by querying the relevant columns in the tables mentioned above. You can also copy the query directly from the report interface for further investigation or debugging purposes [see details here](https://slack.com/archives/C02CTC89LAX/p1672327192977409). Additionally, the compiled tests are stored in the same path as any other dbt tests, typically in your project's 
target/compiled
directory. However, if there is a compilation error, you might not find the compiled code, as the process may fail before generating it [more info](https://slack.com/archives/C02CTC89LAX/p1685348122757179).
f
What about the run code ?
@kapa.ai what about in my dev environment ?
k
A message from kapa.ai
In your dev environment, you can still find the compiled code for elementary tests in the same ways as in other environments: - The compiled SQL is available in the 
elementary_test_results
table under the 
test_results_query
column, and in the 
dbt_run_results
table under the 
compiled_sql
column. - The compiled SQL is also shown in the Elementary report for each test, including in dev environments. - Additionally, the compiled SQL files are generated in your project's 
target/compiled
directory, just like with standard dbt tests. If you are not seeing the compiled code or results in your dev environment, make sure you are running Elementary with the correct profile/target. For example, if you are running locally, you may need to use the 
--profile-target
flag to ensure Elementary is using the right environment configuration. This is important because the report and test results are tied to the environment and target you specify during the run. If you use the wrong target, you might not see the expected results or compiled code in your dev environment [see discussion here](https://slack.com/archives/C02CTC89LAX/p1706114463098459). If you encounter issues where the report is empty or you can't find the compiled code, double-check your target and profile settings.
f
@kapa.ai I have a Database Error, does that mean that the compiled code exist ?
k
A message from kapa.ai
A Database Error does not always mean that the compiled code exists. If the error occurs during the compilation phase (for example, a compilation error), the compiled code may not be generated at all, and you won't find it in the compiled directory. However, if the error occurs during execution (after successful compilation), then the compiled code should exist and you can find it in the usual locations, such as the 
target/compiled
directory or in the relevant tables like 
dbt_run_results
or 
elementary_test_results
. For example, in one case, a user encountered a compilation error and was told that, as this is a compilation error, they won't find compiled code. The error was related to a nested field not being supported, and the compilation failed before any SQL was generated [see here](https://slack.com/archives/C02CTC89LAX/p1685348122757179). So, whether the compiled code exists depends on when the error occurred: - Compilation error: Compiled code does not exist. - Database error during execution: Compiled code does exist.
f
@kapa.ai elementary_test_results is empty - i have installed elementary package but not the CLI
k
A message from kapa.ai
If your 
elementary_test_results
table is empty after installing the Elementary dbt package but you have not installed or used the CLI, the most likely reason is that you have not yet run any dbt tests after deploying the package. The CLI is not required for the test results to be populated—what matters is that you: 1. Install the Elementary dbt package and run 
dbt deps
. 2. Run 
dbt run --select elementary
to create the necessary tables, including 
elementary_test_results
. 3. Run 
dbt test
to execute your tests. Only after running tests will the 
elementary_test_results
table be populated. If you have run 
dbt test
and the table is still empty, check the following: - Ensure you are running both 
dbt run --select elementary
and 
dbt test
on the same target (i.e., the same database/schema configuration). If they are run on different targets, the test results will not appear in the expected table [see discussion](https://slack.com/archives/C02CTC89LAX/p1684095583307509) [and here](https://slack.com/archives/C02CTC89LAX/p1683764076675589). - Make sure you have actually defined tests in your dbt project. If there are no tests, 
dbt test
will have "Nothing to do" and the table will remain empty [see example](https://slack.com/archives/C02CTC89LAX/p1706612460738369). The CLI is only needed for generating reports or using additional observability features, not for populating the 
elementary_test_results
table itself.
f
@kapa.ai How can I check that I am writing/reading to the correct target ?
k
A message from kapa.ai
To check that you are writing and reading to the correct target in Elementary, you should: 1. Verify the target in your dbt commands: Make sure you are using the same 
--target
(or default target) for both 
dbt run --select elementary
and 
dbt test
. If you use different targets, the tables may be created in one schema/database and the tests/results written to another, leading to empty or missing results [see here](https://slack.com/archives/C02CTC89LAX/p1684095583307509) and [here](https://slack.com/archives/C02CTC89LAX/p1683764076675589). 2. Check the output in your terminal: When you run 
dbt run --select elementary
or 
dbt test
, dbt prints the target database and schema it is using. Review this output to confirm it matches your intended environment [see here](https://slack.com/archives/C02CTC89LAX/p1689364183700999). 3. Inspect your 
profiles.yml
and `dbt_project.yml`: Ensure that the target and schema/database settings are consistent and point to the correct locations for both your dbt project and the Elementary package. If you want to force Elementary to use a specific database or schema, you can set this in your 
dbt_project.yml
under the 
elementary
model configuration [see here](https://slack.com/archives/C02CTC89LAX/p1670857782412889). 4. Use the 
edr monitor debug
command: If you have the CLI installed, you can run 
edr monitor debug
to see which database and schema Elementary is connecting to. This helps confirm that your configuration is correct [see here](https://slack.com/archives/C02CTC89LAX/p1684095583307509). By following these steps, you can ensure that both writing (creating tables, running tests) and reading (generating reports, monitoring) are happening in the correct target. If you still encounter issues, double-check that your commands and configuration files are aligned.
f
@kapa.ai How to debug anomaly detection tests ? I want to see actual values.
k
A message from kapa.ai
To debug anomaly detection tests in Elementary and see the actual values: 1. Check the data_monitoring_metrics table: This table stores the historical metric values for each anomaly test. You can query it to inspect the time series data and see how metrics change over time. For example:
Copy code
SELECT        metric_timestamp,        metric_value,        COUNT(*) as metrics_per_bucket    FROM your_schema.data_monitoring_metrics    WHERE table_name = 'your_table'    GROUP BY metric_timestamp, metric_value    ORDER BY metric_timestamp DESC;
This helps you verify if metrics are being collected correctly and if there are any gaps in your data [Anomaly Tests Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). 2. Check the metrics_anomaly_score view: This view shows how anomalies are calculated, including the metric value, training average, standard deviation, z-score, and severity. For example:
Copy code
SELECT        metric_name,        metric_value,        training_avg,        training_stddev,        zscore,        severity    FROM your_schema.metrics_anomaly_score    WHERE table_name = 'your_table'    ORDER BY detected_at DESC;
This allows you to see the actual values used in the anomaly detection logic and understand why a test failed or passed [Anomaly Tests Troubleshooting](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). 3. Elementary report: Generating the Elementary report provides visualizations and graphs of the metrics and anomalies, making it easier to spot issues and understand test failures. The report also includes the compiled SQL and sample data for each test, which is useful for debugging [see here](https://slack.com/archives/C02CTC89LAX/p1680365427971959). 4. elementary_test_results table: This table contains the test results and the 
test_results_query
column, which holds the actual query used for the test. You can use this to see the exact SQL and the data it returned [see here](https://slack.com/archives/C02CTC89LAX/p1658915396893639). By querying these tables and views, and reviewing the report, you can see the actual values and debug your anomaly detection tests effectively.
f
@kapa.ai On Elementary OSS, what do I need to run to update Elementary table and then send the report/Slack alerts
k
A message from kapa.ai
To update the Elementary tables and then send the report or Slack alerts in Elementary OSS, you should follow this sequence: 1. Run the Elementary dbt models to update the tables:
Copy code
dbt run --select elementary
This creates or updates the Elementary tables and views needed for observability and alerting. 2. Run your dbt tests:
Copy code
dbt test
This populates the test results, which Elementary will use for reports and alerts. 3. Send Slack alerts (optional):
Copy code
edr monitor --slack-token <your_slack_token> --slack-channel-name <slack_channel_to_post_at>
Or, if you have configured your Slack integration in 
config.yml
, simply run:
Copy code
edr monitor
This command reads from the test results and sends alerts to Slack if there are failures or anomalies [Setup Slack Alerts](https://docs.elementary-data.com/oss/guides/alerts/send-slack-alerts). 4. Generate and send the report (optional):
Copy code
edr report
This generates the HTML observability report. Or, to send the report to Slack, S3, or GCS:
Copy code
edr send-report
[CLI commands and usage](https://docs.elementary-data.com/oss/cli-commands) Note: - You do not need to run 
dbt run --select elementary
every time unless you have changed your dbt project or upgraded the Elementary package. Most users run it after project changes or merges, and otherwise just run 
dbt test
before generating reports or sending alerts [see here](https://slack.com/archives/C02CTC89LAX/p1659547684760609). - Make sure all commands use the same dbt target/profile to ensure data is written and read from the correct schema. This sequence ensures your Elementary tables are up to date and that your reports and alerts reflect the latest test results.
f
@kapa.ai Can I exclude some filters from the edr monitor command ?
k
A message from kapa.ai
Based on the available knowledge sources, Elementary’s edr monitor command supports filtering alerts by tag, owner, model, status, or resource type using the --filters flag. However, there is no explicit mention in the documentation or community discussions about the ability to exclude (negate) specific filters (e.g., "exclude tag:foo") directly in the command syntax. The documentation only describes how to include alerts matching certain filters, not how to exclude them [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If you need to exclude certain alerts, you may need to structure your filters to only include what you want, rather than explicitly excluding. If exclusion is a critical requirement, it is not currently documented as a supported feature.
f
@kapa.ai What’s the difference between edr send-report and edr monitor ?
k
A message from kapa.ai
The difference between 
edr send-report
and 
edr monitor
is: - 
edr monitor
: Sends Slack alerts on failed tests or model runs according to your configuration (such as channels, owners, tags, etc.). It is focused on real-time alerting for issues detected in your dbt jobs. - 
edr send-report
: Generates a full observability report and uploads it to an external platform (such as Slack, S3, or GCS). It can also send a summary of the report to Slack. This command is used for sharing a comprehensive view of your data health, not just alerts. In summary: - Use 
edr monitor
for sending alerts about failures. - Use 
edr send-report
to share the full report or a summary. This distinction is confirmed in the community and documentation: > • 
edr report
- generates a report on your local machine. > • 
edr send-report
- generates a report and uploads it to a platform of your choice (S3/GCS/Slack). It can also send a summary of the report to Slack. > • 
edr monitor
- sends Slack alerts on failed tests/runs according to the config in your yaml files (channels, owners, tags, etc.) [see here](https://slack.com/archives/C02CTC89LAX/p1689284395118479).
f
@kapa.ai does elementary support dbt singular tests ?
k
A message from kapa.ai
Yes, Elementary fully supports dbt singular tests. Singular (custom SQL) tests are included in the Elementary report and can trigger alerts, just like other dbt tests. You can add configuration such as tags and owners to these tests using the 
config
block, and their results will be visible in the report and included in alerting workflows. However, only singular tests that reference a single model will be linked to that model in the report; tests referencing multiple models may not appear in the lineage view for those models [FAQ](https://docs.elementary-data.com/oss/general/faq) [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1676310474111229) [Custom SQL Tests](https://docs.elementary-data.com/features/data-tests/custom-sql-tests).
f
@kapa.ai Can I have source freshness results in the edr report alert on slack ?
k
A message from kapa.ai
As of the latest information in the knowledge sources, dbt source freshness results are not included in the Elementary report or its Slack summary. They are only supported in Slack alerts sent by 
edr monitor
when a freshness failure occurs, but not in the report or the report summary sent via 
edr send-report
 [see discussion](https://slack.com/archives/C02CTC89LAX/p1689088826025229) [and here](https://slack.com/archives/C02CTC89LAX/p1682416734937579). There is an open (and now closed as implemented) feature request to add dbt source freshness results to the report and its UI, but the knowledge sources do not confirm general availability of this feature yet [see issue](https://github.com/elementary-data/elementary/issues/450). For now, you can receive source freshness alerts in Slack via 
edr monitor
, but not as part of the report summary or report alert. If you need source freshness results in the report or its Slack alert, you may want to follow updates on this feature in the Elementary changelog or GitHub issues.
f
@kapa.ai How can i add a description to elementary.column_anomalies? Will it appear in test notification on slack?
k
A message from kapa.ai
New message text here
f
@kapa.ai What is detection_period ?
k
A message from kapa.ai
detection_period
is a configuration parameter in Elementary anomaly detection tests that defines the time window in which the test looks for anomalies. Only data points within the detection period can be flagged as anomalous. For example, if you set 
detection_period
to 2 days, only data from the last 2 days will be checked for anomalies. You can configure it like this:
Copy code
yaml models:   - name: this_is_a_model     tests:       - elementary.volume_anomalies:           detection_period:             period: day             count: 30
- Supported periods: hour, day, week, month - Default: 2 days For incremental models, the detection period also determines the period for which metrics will be recalculated (to monitor recent backfills). The 
detection_period
parameter only works for tests that have a 
timestamp_column
configured [Detection Period Documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period).
f
What about training_period ?
@kapa.ai what about training_period
k
A message from kapa.ai
training_period
is a configuration parameter in Elementary anomaly detection tests that defines the historical time window used to calculate the "normal" behavior of your data. The values from the training period are used to compute the expected range for the monitored metric. When the test runs, it compares the values in the detection period to those in the training period to determine if any anomalies exist. You can configure it like this:
Copy code
yaml models:   - name: this_is_a_model     tests:       - elementary.volume_anomalies:           training_period:             period: day             count: 30
- Supported periods: day, week, month - Default: 14 days - Works only for tests with a 
timestamp_column
configured. How it works: - For regular tables and views, the full training period is recalculated on each run. - For incremental models and sources, the full training period is calculated on the first run and on full refresh; subsequent runs only calculate the detection period. - If you configure seasonality (e.g., 
day_of_week
), Elementary may automatically increase the training period to ensure enough data points for each seasonality bucket. Impact: - Increasing 
training_period
makes the test less sensitive (fewer false positives, but less likely to catch subtle anomalies). - Decreasing it makes the test more sensitive (more likely to catch anomalies, but also more false positives). For more details, see the [training_period documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period).
f
@kapa.ai What is the default values for time_bucket, anomaly_sensitivity, and anomaly_direction?
k
A message from kapa.ai
The default values for these Elementary anomaly detection test parameters are: - time_bucket: Daily buckets (
period: day, count: 1
) [See documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket) - anomaly_sensitivity: 3 [See documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-sensitivity) - anomaly_direction: 
both
(for most tests), but for freshness and event freshness anomalies, the default is 
spike
[See documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/anomaly-direction) and [freshness anomalies](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies) If you need the default for a specific test type, let me know!
f
@kapa.ai what about seasonality ?
k
A message from kapa.ai
The default value for the 
seasonality
parameter in Elementary anomaly detection tests is **none**—seasonality is not applied unless you explicitly configure it. Supported values for 
seasonality
are 
day_of_week
, 
hour_of_day
, and 
hour_of_week
, but if you do not set this parameter, no seasonality adjustment will be made in the anomaly detection logic [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/seasonality).
f
@kapa.ai can i sent owners for freshness tests ?
k
A message from kapa.ai
Yes, you can set owners for freshness tests in Elementary by adding the 
owner
attribute in the 
meta
field of your source YAML file. However, according to user reports, even when the 
meta.owner
is set for a source table, the owner field may not appear in the Slack alert for source freshness tests. This is a known limitation, and the owner/subscriber fields are not always populated in source freshness alerts, even though the configuration is supported in the YAML [see discussion](https://slack.com/archives/C02CTC89LAX/p1695821762222709). So, while you can configure owners for freshness tests, they may not currently be tagged in Slack alerts for source freshness failures.
f
@kapa.ai Can I use filters to exclude some models from Elementary edr monitor ?
k
A message from kapa.ai
You cannot directly use filters to exclude models in the 
edr monitor
command; the filters (
--filters
or 
--select
) are designed to include alerts for specific tags, owners, models, status, or resource types, not to exclude them. There is no documented support for a "not" or exclusion operator in the filter syntax. To achieve exclusion, you would need to structure your filters to only include the models you want, rather than explicitly excluding certain ones [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If explicit exclusion is critical for your workflow, this feature is not currently supported according to the available documentation and community discussions.
f
@kapa.ai how can I check volume anomalies by a dimension ?
k
A message from kapa.ai
To check volume anomalies by a dimension in Elementary, you should use the 
elementary.dimension_anomalies
test. This test monitors the row count (volume) grouped by the specified dimension(s) over time and alerts on unexpected changes in the distribution. It is best used on low-cardinality fields. Here’s an example configuration:
Copy code
yaml models:   - name: your_model     config:       elementary:         timestamp_column: your_timestamp_column     tests:       - elementary.dimension_anomalies:           dimensions:             - your_dimension_column           time_bucket:             period: day             count: 1
This will monitor the row count for each value of 
your_dimension_column
in daily buckets and detect anomalies in their distribution over time [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/dimension-anomalies). Note: - The 
dimension_anomalies
test only supports row count (volume) per dimension, not other metrics like sum or average per dimension [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1671712508592539). - If you want to monitor volume anomalies for combinations of dimensions, you can list multiple columns under 
dimensions
. If you want to monitor each dimension separately, define separate tests for each [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1668437656150669).
f
@kapa.ai is it default ?
Copy code
time_bucket:
            period: day
            count: 1
k
A message from kapa.ai
Yes, the default value for 
time_bucket
in Elementary anomaly detection tests is:
Copy code
yaml time_bucket:   period: day   count: 1
If you do not specify the 
time_bucket
parameter, Elementary will use daily buckets by default (
period: day
, 
count: 1
) [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/time-bucket).
f
@kapa.ai anomaly on percentage change ?
k
A message from kapa.ai
Elementary does not natively support anomaly detection specifically on the percentage change of a metric as a built-in test type. However, you can approximate this behavior using the 
ignore_small_changes
parameter in anomaly detection tests (such as 
volume_anomalies
, 
column_anomalies
, etc.). This parameter allows you to set thresholds for the minimum percentage increase (
spike_failure_percent_threshold
) or decrease (
drop_failure_percent_threshold
) required to trigger an anomaly, effectively filtering out small percentage changes and only alerting on significant ones. For example:
Copy code
yaml tests:   - elementary.volume_anomalies:       ignore_small_changes:         spike_failure_percent_threshold: 10  # Only alert if increase is 10% or more         drop_failure_percent_threshold: 10   # Only alert if decrease is 10% or more
This does not directly monitor the percentage change itself, but it ensures that only changes above a certain percentage are considered anomalous [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/ignore_small_changes). If you need to monitor the percentage change between two tables or custom metrics, you would need to create a custom model or view that calculates the percentage change, and then apply an anomaly detection test (such as 
column_anomalies
with a 
min
or 
max
monitor) on that result [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1677193288425909). If you are looking for a built-in "anomaly on percentage change" test, the knowledge sources do not indicate that this is currently available.
f
@kapa.ai I need to monitor the percentage change based on two tables
k
A message from kapa.ai
To monitor the percentage change based on two tables in Elementary, there is no native, built-in test that directly supports this use case. However, a recommended workaround is to create a view or model that calculates the percentage change between the two tables, and then apply an Elementary anomaly detection test (such as 
column_anomalies
with a 
min
or 
max
monitor) on the result. This approach is confirmed in community discussions, where it is suggested to: - Define a view that computes the metric (e.g., the ratio or percentage change between the row counts of two tables). - Use 
elementary.column_anomalies
on the calculated column in that view to detect drops or spikes. For example, if you want to monitor the ratio of the row count of 
table1
to 
table2
, you could: 1. Create a view:
Copy code
sql    CREATE VIEW my_schema.table_ratio AS    SELECT      (SELECT COUNT(*) FROM table1) * 1.0 / NULLIF((SELECT COUNT(*) FROM table2), 0) AS ratio
2. Add an Elementary test:
Copy code
yaml    models:      - name: table_ratio        tests:          - elementary.column_anomalies:              column_anomalies:                - min                - max
This will allow you to monitor for anomalies in the calculated percentage change or ratio between the two tables [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1677193288425909). Note: Elementary does not currently support direct anomaly detection across two tables without this workaround. If you need more advanced or native support, you may want to follow updates from the Elementary team, as this feature has been discussed for future releases.
f
HOw much data do i need to make it work ? Elementary says that there is not enough data
@kapa.ai HOw much data do i need to make it work ? Elementary says that there is not enough data
k
A message from kapa.ai
To make Elementary anomaly detection tests work, you generally need at least 14 data points (by default) for the training period. This means: - If you use a `timestamp_column`: You need 14 time buckets (e.g., 14 days if your bucket is daily) with data in your table. If you have enough historical data, Elementary can start detecting anomalies immediately after the first run [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/training-period) and [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1690997004224019). - If you do NOT use a `timestamp_column`: You need to run the test at least 14 times on different days, because each run creates a new data point for the training set [see troubleshooting guide](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/Anomaly-troubleshooting-guide). If you see the message "Not enough data to calculate anomaly score," it usually means you haven't yet accumulated enough data points for the training period. You can reduce the required number of data points by setting the 
min_training_set_size
parameter, but this may make the test more sensitive to outliers and increase false positives [see here](https://slack.com/archives/C02CTC89LAX/p1662960268067179). Summary: - Default: 14 data points (days, weeks, etc. depending on your time bucket) - Can be reduced with 
min_training_set_size
- For non-timestamp tests, you need 14 separate test runs If you still see the error after meeting these requirements, check that your data actually covers the expected time range and that your configuration matches your data's structure.
f
@kapa.ai Does this work ? it doesn't seem to filter alerts? edr monitor --excludes models:bbo
k
A message from kapa.ai
A message from kapa.ai
No, the command 
edr monitor --excludes models:bbo
does not work because Elementary does not support an 
--excludes
flag or any exclusion operator in its filtering syntax. The available filters (
--filters
or previously 
--select
) are only for including alerts by tag, owner, model, status, or resource type, not for excluding them. There is no documented way to exclude models directly in the 
edr monitor
command [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If you need to filter out certain models, you must structure your filters to only include the models you want, rather than explicitly excluding some. Exclusion is not currently supported according to the available documentation and community discussions.
No, the command 
edr monitor --excludes models:bbo
will not work. According to the available documentation and community discussions, Elementary’s 
edr monitor
command does not support an 
--excludes
flag or any direct exclusion syntax for filters. The filtering options (
--filters
or 
--select
) are designed to include specific tags, owners, models, statuses, or resource types, but there is no documented way to exclude models or other entities using a "not" or exclusion operator in the filter syntax. If you need to filter out certain models, you must structure your filters to only include the models you want, rather than explicitly excluding some [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If explicit exclusion is critical for your workflow, this feature is not currently supported.
A message from kapa.ai
No, the command 
edr monitor --excludes models:bbo
does not work. Elementary does not support an 
--excludes
flag or any exclusion operator in its filtering syntax for the 
edr monitor
command. The available filters (
--filters
or previously 
--select
) are only for including alerts by tag, owner, model, status, or resource type, not for excluding them. There is no documented way to exclude models directly in the 
edr monitor
command [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If you need to filter out certain models, you must structure your filters to only include the models you want, rather than explicitly excluding some. Exclusion is not currently supported according to the available documentation and community discussions.
A message from kapa.ai
No, the command 
edr monitor --excludes models:bbo
does not work. Elementary does not support an 
--excludes
flag or any exclusion operator in its filtering syntax for the 
edr monitor
command. The available filters (
--filters
or previously 
--select
) are only for including alerts by tag, owner, model, status, or resource type, not for excluding them. There is no documented way to exclude models directly in the 
edr monitor
command. If you need to filter out certain models, you must structure your filters to only include the models you want, rather than explicitly excluding some [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags).
f
@kapa.ai how does airflow + elementary work ? Can I get info about failed jobs for instance ? In OSS ?
k
A message from kapa.ai
Elementary OSS integrates with Airflow by allowing you to run Elementary commands (such as 
edr monitor
or 
edr report
) as tasks within your Airflow DAGs. This setup enables you to monitor your dbt runs and tests, and send alerts or generate reports as part of your data pipeline orchestration. How it works: - You install the Elementary dbt package and CLI in your Airflow environment (e.g., in a Docker container or KubernetesPodOperator). - After your dbt tasks (such as 
dbt run
and 
dbt test
), you add a task to run Elementary commands. For example, you might run 
edr monitor
to send Slack alerts or 
edr report
to generate an HTML report. - You can configure Airflow to pass job metadata (like job name, job run ID, etc.) to Elementary using environment variables or dbt vars, so that job context is available in the Elementary UI and tables [see guide](https://docs.elementary-data.com/oss/guides/collect-job-data). Failed jobs info: - Elementary can collect and store job metadata from Airflow, including job name, job ID, and job run URL, by capturing environment variables or dbt vars during execution. This information is stored in the 
dbt_invocations
table and can be viewed in the Elementary UI lineage node info tab [see documentation](https://docs.elementary-data.com/cloud/integrations/transformation-and-orchestration/airflow) and [collect job data guide](https://docs.elementary-data.com/oss/guides/collect-job-data). - However, as of now, Elementary OSS does not include job failure information directly in Slack alerts or reports. The job metadata is available for context and troubleshooting, but alerts are focused on dbt test/model failures, not Airflow job/task failures themselves [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1700642681686599). Summary: - You can run Elementary as part of your Airflow DAGs in OSS. - You can collect and view job metadata (job name, run ID, URL) in the Elementary UI and tables. - Alerts and reports focus on dbt test/model failures, not Airflow job failures, but job context is available for troubleshooting. If you need direct alerting on Airflow job failures, you would need to configure that in Airflow itself, not via Elementary OSS.
f
@kapa.ai How to use the --filters flag ? If I want to exclude a folder, can I use syntax like staging.abc ? like in dbt ?
k
A message from kapa.ai
A message from kapa.ai
The 
--filters
flag in Elementary is used to include alerts for specific tags, owners, models, status, or resource types. The syntax is:
Copy code
edr monitor --filters tags:critical edr monitor --filters resource_types:model --filters tags:finance,marketing
You can use multiple 
--filters
flags, and multiple values for the same filter are separated by commas (logical OR). Multiple filters are combined with logical AND. However, there is no support for exclusion (negation) in the filter syntax—so you cannot use 
--filters
to exclude a folder or model, only to include specific ones [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). Regarding folder syntax: You can filter by model name, tag, owner, status, or resource type, but there is no documentation confirming that you can use dbt-style path selectors (like 
staging.abc
or folder paths) with the 
--filters
flag. The filters are not the same as dbt's native selectors, and Elementary's documentation does not mention support for folder-based or path-based filtering in the OSS CLI. Summary: - Use 
--filters
to include alerts for specific models, tags, etc. - You cannot use 
--filters
to exclude folders or models. - Folder/path syntax (like 
staging.abc
) is not documented as supported for filtering in Elementary OSS. If you need to exclude a folder, you would need to structure your filters to only include the models you want, rather than explicitly excluding some.
The 
--filters
flag in Elementary is used to include alerts for specific tags, owners, models, status, or resource types. The syntax is:
Copy code
edr monitor --filters tags:critical edr monitor --filters tags:finance,marketing edr monitor --filters resource_types:model --filters tags:finance,marketing
You can use multiple 
--filters
flags, and multiple values for the same filter are separated by commas (logical OR). Multiple filters are combined with logical AND [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). Regarding folder exclusion: Elementary’s filtering does not support direct exclusion (negation) of folders or models. There is no documented way to use a "not" or exclusion operator in the filter syntax. The filters are only for inclusion. If you want to filter by a folder (like 
staging.abc
), you can use the model filter to include models from that folder, but you cannot use it to exclude them [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). Example for including models in a folder:
Copy code
edr monitor --filters models:staging.abc
This would include only models in the 
staging.abc
folder, not exclude them. Summary: - Use 
--filters
to include, not exclude. - No support for exclusion or negation in filter syntax. - You can filter by model or folder for inclusion, but not for exclusion. If you need to exclude a folder, you would need to structure your filters to only include what you want, rather than explicitly excluding certain folders or models.
A message from kapa.ai
The 
--filters
flag in Elementary is used to include alerts for specific tags, owners, models, status, or resource types. The syntax is:
Copy code
edr monitor --filters tags:critical edr monitor --filters models:my_model edr monitor --filters tags:finance,marketing
You can use multiple 
--filters
flags, and multiple values for the same filter are separated by commas (logical OR). Multiple filters are combined with logical AND. However, there is no support for exclusion (negation) in the filter syntax—so you cannot use 
--filters
to exclude a folder or model, only to include specific ones. There is also no documentation confirming that you can use dbt-style path selectors (like 
staging.abc
) or folder-based filtering with the 
--filters
flag in Elementary OSS [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). If you need to exclude a folder, you would need to structure your filters to only include the models you want, rather than explicitly excluding some.
f
@kapa.ai How I can define tags ?
k
A message from kapa.ai
A message from kapa.ai
You can define tags in your dbt project to provide context and segment your data assets by business domains, data products, priority, etc. In Elementary, tags are automatically included in alerts and can be used to filter views and distribute alerts to different channels. How to define tags: - At the model or folder level: You can add tags in your 
dbt_project.yml
file to entire directories or specific models:
Copy code
yaml   models:     analytics:       marketing:         +tags: marketing         public:           +tags: marketing-public
This will apply the 
marketing
tag to all models in the 
marketing
folder and 
marketing-public
to those in the 
public
subfolder. - At the model level in a YAML file:
Copy code
yaml   models:     - name: return_on_ad_spend       tags:         - marketing-public         - marketing
- At the test level: You can add tags to tests using the 
tags
key:
Copy code
yaml   tests:     - not_null:         tags: [completeness]
Note: Tags are aggregated, so if a specific model under a directory has a different tag, the model will have both tags. It’s recommended to leverage dbt directories hierarchy to set tags to entire directories for easier management [see best practices](https://docs.elementary-data.com/best-practices/governance-for-observability). Usage in Elementary: Tags are included in alerts, can be used for filtering in the UI, and for routing alerts to different channels [see documentation](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alert-properties-in-yml-files).
You can define tags in Elementary by adding them to your dbt model, source, or test configurations using the 
tags
key. Here are the main ways to define tags: ### 1. Model-level tags Add tags in your model’s YAML file, either directly under the model or within the 
config
block:
Copy code
yaml models:   - name: my_model     config:       tags: ["marketing", "critical"]
or
Copy code
yaml models:   - name: my_model     tags: ["marketing", "critical"]
Both approaches are supported, but using the 
config
block is recommended for consistency with dbt best practices [see example](https://slack.com/archives/C02CTC89LAX/p1678370484105529). ### 2. Folder-level tags (dbt_project.yml) You can tag all models in a folder by adding tags in your `dbt_project.yml`:
Copy code
yaml models:   analytics:     marketing:       +tags: marketing     public:       +tags: marketing-public
This will apply the tag to all models in the specified folder [see best practices](https://docs.elementary-data.com/best-practices/governance-for-observability). ### 3. Source-level tags For sources, tags should be placed under the 
meta
key:
Copy code
yaml sources:   - name: source_name     tables:       - name: source_table         meta:           tags: ["tag_1", "tag_2"]
This ensures the tags are recognized by Elementary and appear in reports [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1680187693589419). ### 4. Test-level tags While you can add tags to tests using the 
tags
key, currently Elementary only supports model-level tags in the report UI. Test-level tags may appear in Slack alerts but not in the report filters [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1680160045323839).
Copy code
yaml tests:   - not_null:       tags: ["completeness"]
Note: Tags are aggregated, so if a model and its folder both have tags, the model will have both tags. For more details, see the [Elementary documentation on tags](https://docs.elementary-data.com/best-practices/governance-for-observability).
A message from kapa.ai
You can define tags in your dbt project for use with Elementary by adding the 
tags
key at the model, folder, or test level. At the model or folder level (in 
dbt_project.yml
):
Copy code
yaml models:   analytics:     marketing:       +tags: marketing     public:       +tags: marketing-public
This applies the 
marketing
tag to all models in the 
marketing
folder, and 
marketing-public
to those in the 
public
subfolder. Tags are aggregated, so a model can inherit tags from its folder and also have its own tags defined directly on the model [see best practices](https://docs.elementary-data.com/best-practices/governance-for-observability). At the model level (in a YAML file):
Copy code
yaml models:   - name: return_on_ad_spend     tags:       - marketing-public       - marketing
At the test level:
Copy code
yaml tests:   - not_null:       tags: [completeness]
However, currently, Elementary primarily uses model-level tags for filtering and alerting in the UI and reports. Test-level tags may appear in Slack alerts, but not in the report filters [see discussion](https://slack.com/archives/C02CTC89LAX/p1680160045323839). Note: - Tags can be used to segment your data assets, filter views, and route alerts to different channels in Elementary. - It’s recommended to leverage the dbt directory hierarchy to set tags at the folder level for easier management. For more details, see the [Elementary documentation on tags](https://docs.elementary-data.com/best-practices/governance-for-observability).
f
@kapa.ai Can you explain the detection_period parameter ? What's the default value ?
k
A message from kapa.ai
The 
detection_period
parameter in Elementary anomaly detection tests defines the time window in which the test looks for anomalies. Only data points within this detection period are checked and can be flagged as anomalous. For example, if you set 
detection_period
to 2 days, only data from the last 2 days will be evaluated for anomalies. You can configure it like this:
Copy code
yaml models:   - name: this_is_a_model     tests:       - elementary.volume_anomalies:           detection_period:             period: day             count: 30
- Supported periods: hour, day, week, month - Default value: 2 days For incremental models, the detection period also determines the period for which metrics will be recalculated, which is useful for monitoring recent backfills. The 
detection_period
parameter only works for tests that have a 
timestamp_column
configured [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/detection-period).
f
@kapa.ai How can I delete a model from the UI/alerts? I deleted it in dbt but it's still appearing.
k
A message from kapa.ai
If you have deleted a model from your dbt project but it is still appearing in the Elementary UI or alerts, this is likely because the artifact tables (such as 
dbt_models
) reflect the current state of your project only after you run your dbt commands. You should run your dbt project again (e.g., 
dbt run
and 
dbt run --select elementary
) to update the artifact tables and remove references to deleted models. Once you do this, the deleted model should no longer appear in the UI or trigger alerts [see discussion](https://slack.com/archives/C02CTC89LAX/p1737364638217529). If the model still appears after running these commands, ensure you are using the correct target and that your dbt and Elementary runs are pointed at the same schema/database.
f
@kapa.ai I am running Elementary OSS with Docker. However i am hitting a request limit from Docker. How can I add a login step to Docker to get a higher limit ?
k
A message from kapa.ai
I'm sorry, but the knowledge sources do not contain information about adding a Docker login step to increase request limits when running Elementary OSS with Docker. The available documentation and community discussions focus on using the Elementary Docker image and running Elementary commands, but do not address Docker authentication or rate limiting. If you need guidance on this topic, you may want to consult Docker's official documentation or support channels.
f
@kapa.ai Can I get debug for a model run fail in elementary OSS ?
k
A message from kapa.ai
In Elementary OSS, you can get some debug information for a model run failure, but there are limitations: - The Elementary report and UI will show the status (Success/Error) of model runs and include the error message in the alert if you have Slack alerts configured. However, the full error message and detailed debug information are **not included in the OSS report UI**—they are only available in the alert payload or in the dbt logs [see Slack discussion](https://slack.com/archives/C02CTC89LAX/p1736754082968199). - The new incident management and detailed error inspection features (such as viewing the full dbt model build error message in the UI) are available in the Elementary Cloud version, not in OSS [see changelog](https://docs.elementary-data.com/changelog#a-better-way-to-triage-test-and-model-failures). Workarounds in OSS: - You can check the regular dbt logs for detailed error output. These logs will contain the full stack trace and error context for failed model runs. - The Elementary report will show the model run status and, if configured, send the error message in Slack alerts, but not the full debug context. Summary: You can see that a model run failed and get the error message in alerts, but for full debug details, you need to check your dbt logs. The OSS UI does not provide full debug/error inspection for model run failures.
f
@kapa.ai Can I used edr monitor --excludes With multiple tags ?
k
A message from kapa.ai
No, the 
edr monitor --excludes
command does not work in Elementary, whether with single or multiple tags. Elementary does not support an 
--excludes
flag or any exclusion operator in its filtering syntax for the 
edr monitor
command. The available filters (
--filters
or previously 
--select
) are only for including alerts by tag, owner, model, status, or resource type, not for excluding them. There is no documented way to exclude models, tags, or other entities directly in the 
edr monitor
command [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). What you can do instead: Structure your filters to only include the tags you want, rather than trying to exclude certain tags. For example:
Copy code
edr monitor --filters tags:tag1,tag2
This will send alerts only for models with 
tag1
or 
tag2
, effectively excluding all other tags. If explicit exclusion is critical for your workflow, this feature is not currently supported according to the available documentation and community discussions.
f
@kapa.ai elementary anomaly detection, can i use two dimensions?
k
A message from kapa.ai
Yes, you can use multiple dimensions in Elementary anomaly detection tests. However, it's important to understand how they work together: For 
dimension_anomalies
test: When you configure multiple dimensions, the test monitors the combination of those dimensions together, not each dimension separately. For example:
Copy code
yaml tests:   - elementary.dimension_anomalies:       dimensions:         - hardware_platform         - app_version
This will check the row count for each unique combination of 
hardware_platform
and 
app_version
(e.g., "android; 5.15.5.3393") [see discussion](https://slack.com/archives/C02CTC89LAX/p1690470550072969). To monitor each dimension separately: If you want to monitor each dimension independently, you should configure separate tests for each dimension:
Copy code
yaml tests:   - elementary.dimension_anomalies:       dimensions:         - hardware_platform   - elementary.dimension_anomalies:       dimensions:         - app_version
[See discussion](https://slack.com/archives/C02CTC89LAX/p1668437656150669) and [another example](https://slack.com/archives/C02CTC89LAX/p1698160773464849). For 
column_anomalies
test: The same logic applies. You can add multiple dimensions to segment your column metrics:
Copy code
yaml columns:   - name: in_app_purchase_revenue     tests:       - elementary.column_anomalies:           column_anomalies:             - sum           dimensions:             - app_name             - country
This will calculate the sum of 
in_app_purchase_revenue
for each combination of 
app_name
and 
country
[see changelog](https://docs.elementary-data.com/changelog#column-anomalies-by-dimensions). Best practice: Use low-cardinality fields as dimensions to avoid creating too many combinations [see documentation](https://docs.elementary-data.com/data-tests/anomaly-detection-configuration/dimensions).
f
@kapa.ai I have some dev jobs that are running in my production environment on dbt. However Elementary is also reporting on those. Can I exclude them somehow ?
k
A message from kapa.ai
Based on the knowledge sources, there are several ways you can exclude dev jobs running in your production environment from Elementary reporting: ## 1. Disable Elementary for specific runs using environment variables You can disable Elementary's on-run-end hooks for specific dbt runs by setting the 
edr_cli_run
variable. This prevents Elementary from collecting data for those runs:
Copy code
bash dbt run --vars 'edr_cli_run: true'
This approach is useful for ad-hoc or dev runs that you don't want Elementary to track [see documentation](https://docs.elementary-data.com/dbt/on-run-end_hooks#can-i-disable-the-on-run-end-hooks). ## 2. Use tags to filter alerts If your dev jobs run specific models or tests, you can: 1. Tag those models/tests with a specific tag (e.g., 
dev
or 
experimental
) 2. Use the 
--filters
flag in 
edr monitor
to exclude them by only including the tags you want:
Copy code
bash edr monitor --filters tags:production,critical
This will only send alerts for models with the 
production
or 
critical
tags, effectively excluding your dev-tagged models [Alerts CLI flags](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags). ## 3. Disable specific hooks for dev runs You can conditionally disable Elementary's data collection hooks based on environment variables or dbt vars:
Copy code
yaml vars:   disable_run_results: "{{ env_var('IS_DEV_RUN', 'false') == 'true' }}"   disable_tests_results: "{{ env_var('IS_DEV_RUN', 'false') == 'true' }}"   disable_dbt_artifacts_autoupload: "{{ env_var('IS_DEV_RUN', 'false') == 'true' }}"   disable_dbt_invocation_autoupload: "{{ env_var('IS_DEV_RUN', 'false') == 'true' }}"
Then set 
IS_DEV_RUN=true
when running your dev jobs [see discussion](https://slack.com/archives/C02CTC89LAX/p1669760137760489). Note: Elementary does not support an 
--excludes
flag or direct exclusion syntax in the 
edr monitor
command, so you need to use inclusion-based filtering or disable data collection for specific runs [see discussion](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-cli-flags).
f
can you find messages from users ?
@kapa.ai can you find messages from users ?
k
A message from kapa.ai
I'm not sure what you mean by "find messages from users." Could you clarify what you're looking for? If you're asking whether Elementary can: - Display or search user messages/comments within the Elementary UI or reports: The knowledge sources do not indicate that Elementary has a built-in messaging or commenting system for users. - Track which users are running queries or dbt jobs: Elementary can collect query logs and job metadata, including the user performing the query, when integrated with your data warehouse [see security and privacy](https://docs.elementary-data.com/cloud/general/security-and-privacy#what-information-does-elementary-collect%3F). - Access Slack messages or community discussions: I can reference discussions from the Elementary Slack community that are included in the knowledge sources, but I cannot search or retrieve live messages from Slack or other platforms. Could you provide more context about what you're trying to achieve? For example, are you looking for information about user activity tracking, alert notifications, or something else?
f
however for CI, it's working fine
@kapa.ai can you find messages from users related to those issues? how do they do it ?
k
A message from kapa.ai
Based on the knowledge sources, I can find several user discussions about how they handle model run failures and visibility in Elementary. Here are the key approaches users have taken: ## 1. Creating custom monitoring models One user created their own model to monitor recurring errors on models and tests. They referenced the 
alerts_dbt_tests
and 
alerts_dbt_models
views to track issues and prioritize them. They wanted to track: - Error recurrence - Time from first error to resolution - Quality improvements over time They used 
adapter.get_relation()
to reference Elementary's internal alert tables:
Copy code
sql {% set elementary_database, elementary_schema = elementary.get_package_database_and_schema() %} {% set alerts_relation = adapter.get_relation(elementary_database, elementary_schema, "alerts") %} select * from {{ alerts_relation }}
Their goal was to create a culture where recurring alerts without action would trigger reevaluation of the model or test [see discussion](https://slack.com/archives/C02CTC89LAX/p1673994083604889). ## 2. Feature requests for better visibility Multiple users requested better visibility of model run failures on the dashboard front page. Currently, they have to: - Go into the 
Model Runs
page - Filter for failed runs - Check each failed model run to see if it was the last attempt that failed or a previous one Users wanted model run results displayed prominently on the dashboard, similar to test results, with circle diagrams and graphs showing pass/fail/warning status [see GitHub issue](https://github.com/elementary-data/elementary/issues/689) and [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1676621045227169). ## 3. Current workarounds Users currently: - Check the 
Model Runs
tab in the Elementary report to see model execution status - Use Slack alerts for model failures (enabled by default in Elementary) - Query the 
dbt_run_results
table directly for model run information The Elementary team acknowledged this gap and noted that the dashboard was designed with data quality (test results) in mind, but agreed that model failures represent quality issues that should be more visible [see discussion](https://slack.com/archives/C02CTC89LAX/p1676621045227169).
f
@kapa.ai if a model fails then succeed, i am still getting a failure alert can i change this ?
k
A message from kapa.ai
Based on the knowledge sources, if a model fails and then succeeds, you may still receive a failure alert because Elementary sends alerts for failures as they occur, even if the issue is later resolved. This is a known behavior, and there are several approaches to address it: ## 1. Use alert suppression interval You can configure an 
alert_suppression_interval
to prevent multiple alerts for the same issue within a specified time period. This won't prevent the initial alert, but it will reduce noise if the same issue recurs:
Copy code
yaml models:   - name: my_model_name     meta:       alert_suppression_interval: 24  # hours
[See documentation](https://docs.elementary-data.com/oss/guides/alerts/alerts-configuration#alerts-distribution) ## 2. Run 
edr monitor
only after retries Several users have implemented a workflow where they: 1. Run dbt models 2. Retry any failed models 3. Only run 
edr monitor
after all retries are complete This way, you only get alerts for models that failed even after retries. However, this means successful runs and passing tests won't be recorded in Elementary for the initial run [see discussion](https://slack.com/archives/C02CTC89LAX/p1670502047073109). ## 3. Planned feature: Alert deduplication The Elementary team has discussed adding a 
--dedup
flag to the CLI that would: - Only send one failure alert per test/model/seed - Not send a failure alert if the test/model/seed succeeded afterwards This feature was discussed but the knowledge sources don't confirm if it's been implemented yet [see discussion](https://slack.com/archives/C02CTC89LAX/p1674118291945399). ## 4. Current limitation As one user noted, if you run a model locally and it fails, then fix it and rerun successfully, you may still get an alert for the original failure because alerts are sent when failures are synced to Elementary, not immediately. The team acknowledged this and mentioned they're working on alert lifecycle improvements to avoid sending alerts if issues are already resolved [see discussion](https://slack.com/archives/C02CTC89LAX/p1690801293721309). Recommendation: The most practical current solution is to run 
edr monitor
only after your retry logic completes, so you only alert on persistent failures.
41 Views
Open in Slack
PreviousNext
Powered by