Yes, Elementary OSS does offer pipeline tests. According to the knowledge sources, Elementary provides two main types of tests: 1. Pipeline tests: These monitor the health of data pipelines, ensuring timely and smooth data ingestion, transformation, and loading. 2. Data quality tests: These validate data accuracy, completeness, and correctness, detect anomalies and schema changes, and ensure the data meets predefined business rules. The Elementary dbt package, which is part of the OSS offering, includes several anomaly detection tests that can be used for pipeline monitoring, such as: - Volume anomalies: Monitors table row count over time to detect drops or spikes in volume - Freshness anomalies: Monitors the latest timestamp of a table to detect data delays - Event freshness anomalies: Monitors the gap between the latest event timestamp and its loading time However, it's worth noting that there are differences between the OSS and Cloud implementations of these tests. In the OSS version, pipeline health monitors for freshness and volume are implemented as dbt tests that run in dbt and need to be manually added in code, whereas in the Cloud version, these are automated and run in the Cloud with out-of-the-box full coverage [Pipeline Health Monitors Comparison](https://docs.elementary-data.com/data-tests/anomaly-detection-tests-oss-vs-cloud). The Elementary Cloud Platform also offers automated pipeline tests that analyze query history metadata instead of querying the dbt tables directly, which is faster and more cost-efficient. For more information about Elementary tests, you can refer to the [Elementary Data Tests documentation](https://docs.elementary-data.com/data-tests/introduction).

To enable pipeline tests in Elementary OSS, you need to add them to your dbt project. Here's how to do it: 1. First, make sure you have [installed the Elementary dbt package](https://docs.elementary-data.com/quickstart#install-the-dbt-package) in your project. 2. Then, add the pipeline tests (such as volume and freshness anomalies) to your models or sources in your dbt YAML files:

yaml models:   - name: your_model_name     tests:       - elementary.volume_anomalies:           severity: 'error'  # or 'warn'           tags:             - elementary       - elementary.freshness_anomalies:           timestamp_column: your_timestamp_column  # Mandatory for freshness tests           severity: 'error'  # or 'warn'           tags:             - elementary

For freshness anomalies, you must specify a

timestamp_column

. As noted in a community question, if you don't provide this parameter, you'll get an error: "Missing mandatory configuration: ['timestamp_column']" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1697098266405579). You can also configure additional parameters for these tests: - `seasonality`: day_of_week, hour_of_day, etc. - `time_bucket`: period and count - `anomaly_sensitivity`: numerical value (default is 3) - `anomaly_direction`: both, up, or down Note that in Elementary OSS, these tests need to be manually added to each model or source you want to monitor, unlike in Elementary Cloud where monitoring is automated and out-of-the-box [Pipeline Health Monitors Comparison](https://docs.elementary-data.com/data-tests/anomaly-detection-tests-oss-vs-cloud). You can also add tests to source tables:

yaml sources:   - name: your_source_name     tables:       - name: your_table_name         tests:           - elementary.volume_anomalies           - elementary.freshness_anomalies:               timestamp_column: your_timestamp_column

After adding these tests, they will run as part of your regular dbt test runs, and the results will be displayed in the Elementary observability report if you're using the Elementary CLI. For more detailed information on configuring these tests, refer to the [Add anomaly detection tests documentation](https://docs.elementary-data.com/data-tests/add-elementary-tests).

Based on the knowledge sources, a table appears as "monitored" in the Elementary UI when it has at least one test associated with it. This is calculated based on the

dbt_tests

and

dbt_models

tables that Elementary creates in your data warehouse. According to a response from an Elementary support staff member: > "It's calculated based on the

dbt_tests

and

dbt_models

tables. We consider a monitored table if its model has at least one test." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1692272312227569) The "Monitored tables" visualization in the Elementary report shows the count of tables that have tests configured for them. It's generated by joining data between

dbt_models

and

dbt_sources

to find all tables, and then joining with

dbt_tests

to see which tables have monitors configured. By default, Elementary's own internal tables are excluded from this count, but tables from other dbt packages might be included. This has been a point of discussion in the community, with some users wanting to exclude other packages' tables from the monitoring count as well. If you want to mark certain tables as "intentionally not monitored" (so they don't appear as problematic unmonitored tables), one suggested approach is to use tags: > "I can suggest adding a tag called

critical

or

monitored

and then you can filter in the dashboard based on this new tag. In dbt you can also apply tags globally in your

dbt_project.yml

so you don't need to manually mark model by model." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1680038056113519) There's also an open GitHub issue (#794) about excluding models based on tags from Elementary reports, which would allow you to exclude certain unmonitored models when sharing reports with stakeholders. To summarize, a table is considered "monitored" if it has at least one test associated with it, and "unmonitored" if it doesn't have any tests. LLM: I'll analyze the knowledge sources to answer your question about monitored vs unmonitored tables in Elementary UI. Based on the information provided, a table appears as "monitored" in the Elementary UI when it has at least one test associated with it. This is calculated based on the

dbt_tests

and

dbt_models

tables that Elementary creates in your data warehouse. According to a response from an Elementary support staff member: > "It's calculated based on the

dbt_tests

and

dbt_models

tables. We consider a monitored table if its model has at least one test." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1692272312227569) The "Monitored tables" visualization in the Elementary report shows the count of tables that have tests configured for them. It's generated by joining data between

dbt_models

and

dbt_sources

to find all tables, and then joining with

dbt_tests

to see which tables have monitors configured. By default, Elementary's own internal tables are excluded from this count, but tables from other dbt packages might be included. This has been a point of discussion in the community, with some users wanting to exclude other packages' tables from the monitoring count as well. If you want to mark certain tables as "intentionally not monitored" (so they don't appear as problematic unmonitored tables), one suggested approach is to use tags: > "I can suggest adding a tag called

critical

or

monitored

and then you can filter in the dashboard based on this new tag. In dbt you can also apply tags globally in your

dbt_project.yml

so you don't need to manually mark model by model." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1680038056113519) There's also an open GitHub issue (#794) about excluding models based on tags from Elementary reports, which would allow you to exclude certain unmonitored models when sharing reports with stakeholders. To summarize, a table is considered "monitored" if it has at least one test associated with it, and "unmonitored" if it doesn't have any tests.

Based on the knowledge sources, there isn't a direct way to add tests to all models at once in Elementary OSS. This is a limitation of dbt itself, not specific to Elementary. According to multiple support staff responses in Slack discussions: > "Unfortunately at the moment it's not possible to define dbt tests globally to run on all tables (not Elementary test and not any other test in dbt)." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1676902538214649) > "It's a dbt limitation." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1696281462988319) There are a few workarounds mentioned in the knowledge sources: 1. YAML Anchors: You can use YAML anchors to make it easier to add the same test to multiple models:

yaml version: 2  elementary_schema_change_test: &elementary_schema_change_test   elementary.schema_changes:     tags: [elementary, elementary_schema_change_tests]     config:       severity: warn  tests: &tests   - *elementary_schema_change_test  models:   - name: my_model     tests: *tests

[Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1704886421763019) 2. Custom Script: One user mentioned they could "write a script to add the tests in bulk" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1689113314530069) 3. Elementary Cloud: The cloud version of Elementary (which is a paid product) does offer a solution for adding tests in bulk: > "Our cloud product (currently in private beta) does offer a solution for it" [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1689113314530069) > "We have bulk tests configuration as a feature in Elementary Cloud, you add it from the UI and it opens a PR to your code." [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1696281462988319) 4. Run specific test types: You can run all Elementary tests using:

dbt test --select package:elementary

[Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1695225726822319) The May 2024 changelog also mentions a new feature in Elementary Cloud: > "Elementary now supports adding any dbt test in bulk and directly from the UI!" [Elementary Changelog](https://docs.elementary-data.com/changelog#add-any-dbt-test-via-ui) In summary, for the open-source version, you need to add tests model-by-model, potentially using YAML anchors to make it more manageable, or create a custom script to automate the process. The bulk test configuration feature is available in Elementary Cloud.