To set up source freshness alerts in Slack with Elementary Data, you need to follow these steps: 1. First, run your dbt source freshness command:

dbt source freshness

2. Then, upload the source freshness results to Elementary:

edr run-operation upload-source-freshness --project-dir .

3. Finally, send the alerts to Slack:

edr monitor --slack-token YOUR_SLACK_TOKEN --slack-channel-name YOUR_CHANNEL --env YOUR_ENV

Note that source freshness results are currently only available in alerts, not in the HTML report. As mentioned in the knowledge sources, "The issue is that

source freshness

is not yet supported in the Elementary report, only in the alerts" [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1689088826025229). You can customize your source freshness alerts by adding meta configurations to your sources in the dbt_project.yml file:

yaml sources:   +meta:     alerts_config:       alert_suppression_interval: 12       slack_group_alerts_by: alert       channel: your-custom-channel

However, be aware that there have been some issues with source alert configurations. According to a support staff member, "In sources dbt priorities for the meta field are the other way around currently - Top priority is project level, then source, then table. This means that project level config overrides source / table level config" [Support staff response](https://slack.com/archives/C02CTC89LAX/p1680099515160789). By default, Elementary doesn't alert on warnings. If you want to receive warnings, you can add this to your dbt_project.yml:

yaml vars:   disable_warn_alerts: false

For more detailed information, you can check the [Elementary documentation on alerting for source freshness failures](https://docs.elementary-data.com/oss/guides/alerts/send-slack-alerts#alert-on-source-freshness-failures).

To send the Elementary Data report summary notification to Slack, you can use the

edr send-report

command with Slack configuration. Here's how to do it: ## For reports shared via Slack attachment

edr send-report --slack-token <SLACK_TOKEN> --slack-channel-name <CHANNEL_NAME>

When sharing the report via Slack, the summary is sent by default along with the HTML attachment [Share via Slack](https://docs.elementary-data.com/oss/guides/share-observability-report/share-via-slack). ## For reports hosted on cloud storage (S3 or GCS) If you're hosting your report on cloud storage, you need to provide Slack credentials to enable the summary: For AWS S3:

edr send-report --aws-profile-name <AWS_PROFILE> --s3-bucket-name <BUCKET_NAME> --slack-token <SLACK_TOKEN> --slack-channel-name <CHANNEL_NAME> --update-bucket-website true

For Google Cloud Storage:

edr send-report --google-service-account-path <SERVICE_ACCOUNT_PATH> --gcs-bucket-name <BUCKET_NAME> --slack-token <SLACK_TOKEN> --slack-channel-name <CHANNEL_NAME>

A link to the hosted report will be included in the summary [Send report summary](https://docs.elementary-data.com/oss/guides/share-observability-report/send-report-summary). ## Send only the summary (without HTML attachment) If you want to send only the summary without attaching the HTML report (which can be large):

edr send-report --slack-token <SLACK_TOKEN> --slack-channel-name <CHANNEL_NAME> --disable html_attachment

## Filtering the summary results You can filter your results summary by tag, owner, or model (only when disabling the HTML attachment):

edr send-report --select tag:finance --disable html_attachment edr send-report --select config.meta.owner:@jeff --disable html_attachment edr send-report --select model:customers --disable html_attachment

Note that as mentioned in our previous conversation, source freshness results are currently only available in alerts, not in the HTML report or summary.

I don't have specific information about report summary notifications in Elementary Cloud in the provided knowledge sources. The documents primarily cover the open-source version of Elementary Data and don't contain details about how report summaries work in the cloud version. In one of the knowledge sources, there is a brief mention that the cloud version offers additional capabilities not available in the open source package: "This is not something that is offered in the open source package. In the cloud version we do and you can do alert-routing and raise incidents etc." [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1729618316203879) This suggests that Elementary Cloud likely has more advanced notification and incident management features, but I don't have the specific details on how to configure report summary notifications in Elementary Cloud. For the most accurate and up-to-date information about Elementary Cloud features, I'd recommend: 1. Checking the official Elementary Cloud documentation 2. Reaching out to Elementary support through their community Slack 3. Contacting your Elementary account representative Would you like me to provide any additional information about the open-source version's report summary capabilities?

I understand you're getting duplicate failure notifications from both dbt and Elementary, and you'd like to consolidate to just use Elementary's notifications. This is a common issue that several users have faced. To use only Elementary for failure notifications and avoid duplication with dbt alerts, you have a few options: ## Option 1: Disable model alerts in Elementary If you're already getting dbt notifications for model failures and only want Elementary for test failures:

yaml vars:   disable_model_alerts: true

Add this to your

dbt_project.yml

file, then run:

dbt run -s alerts_dbt_models

This will disable Elementary's model failure alerts while keeping test alerts [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1682602063978059). ## Option 2: Configure your CI/CD pipeline As one user explained: > "The reason is because dbt cloud already notify us on that, so we don't need two notifications" [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1681291354494619) You can modify your CI/CD pipeline to either: 1. Use only Elementary for all notifications 2. Use dbt for some types and Elementary for others ## Option 3: Use selective alerting with Elementary You can run Elementary with specific selectors to control which alerts you receive:

edr monitor --select 'statuses:warn' -ch '#warn-channel' edr monitor --select 'statuses:fail' -ch '#fail-channel'

This allows you to route different types of alerts to different channels [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1698434621093649). ## Option 4: Use tags to control which alerts are sent You can tag specific models or tests with "elementary" and then use:

edr monitor --select tag:elementary

This will only send alerts for items with that specific tag [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1673893162838399). Remember that after changing any configuration in your

dbt_project.yml

, you'll need to run

dbt run -s elementary

to apply the changes before they take effect. Would you like more specific guidance on any of these approaches?

Yes, you can definitely change the database schema for Elementary between prod and dev environments. This is a common requirement to keep development work separate from production data. Here are several approaches based on the knowledge sources: ## Option 1: Configure different schemas in dbt_project.yml You can conditionally set the schema based on the target environment:

yaml models:   elementary:     +schema: "{{ 'elementary' if target.name == 'prod' else 'dev_elementary' }}"

This will create Elementary tables in different schemas depending on your target [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1668005063898609). ## Option 2: Use different targets in profiles.yml Create separate targets in your profiles.yml file:

yaml elementary:   outputs:     dev:       type: "snowflake"  # or your database type       database: dev_database       schema: dev_elementary       # other connection details     prod:       type: "snowflake"  # or your database type       database: prod_database       schema: prod_elementary       # other connection details   target: dev  # default target

Then specify the target when running Elementary commands: - For development:

edr report --profile-target dev

- For production:

edr report --profile-target prod

[Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1699363455840319) ## Option 3: Disable Elementary in dev If you want to completely disable Elementary in development:

yaml models:   elementary:     +schema: "elementary"     # To disable elementary for dev, uncomment this:     enabled: "{{ target.name in ['prod','analytics'] }}"

This will only run Elementary in production environments [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1699907896171609). ## Important notes: 1. If you have a custom

generate_schema_name

macro in your project, it might override your schema settings. You'll need to adjust your macro to handle Elementary schemas correctly [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1685020975522279). 2. For CLI commands like

edr report

, you need to use

--profile-target

to specify which environment to use, as the

--env

flag is only cosmetic and doesn't affect database connections [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1681336858403709). 3. Remember to run

dbt run --select elementary

after changing configurations to create the Elementary tables in the new schema. Would you like more specific guidance for your particular database or setup?

Yes, you can run tests from Elementary Cloud instead of directly running them through dbt. Elementary Cloud offers a UI-based approach to configuring and running tests while still leveraging the power of dbt tests. Elementary Cloud provides a configuration-as-code approach where you can: 1. Add tests through the UI: You can configure tests directly in the Elementary Cloud interface, which will then create the necessary code in your dbt project. As mentioned in the documentation, "In Elementary Cloud, you can save time by adding tests in bulk from the UI that will be added to your code" [Configuration-as-Code](https://docs.elementary-data.com/features/config-as-code). 2. Create tests without writing code: "You can allow data analysts to create quality tests without writing any code. Elementary will take care of it for them and open pull requests on their behalf" [Configuration-as-Code](https://docs.elementary-data.com/features/config-as-code). 3. Add any dbt test via UI: A recent update (May 2024) added support for "adding any dbt test in bulk and directly from the UI" including support for dbt-expectation and dbt-utils packages [Elementary Changelog](https://docs.elementary-data.com/changelog#add-any-dbt-test-via-ui). To use this feature, you need to: 1. [Connect a code repository](https://docs.elementary-data.com/cloud/integrations/code-repo/connect-code-repo) to your project 2. Choose a target 3. Choose a test 4. Configure the test 5. Review and submit When you configure a test in Elementary Cloud, it will open a pull request/merge request in your code repository [Tests configuration](https://docs.elementary-data.com/data-tests/elementary-tests-configuration). It's important to note that while you're configuring tests through the Elementary Cloud UI, these tests are still ultimately dbt tests that run in your dbt environment. Elementary Cloud is providing a more user-friendly interface for managing these tests rather than replacing dbt entirely. Would you like more information about specific types of tests you can configure through Elementary Cloud?

Based on the knowledge sources provided, I don't have specific information about a feature called "Runs in cloud" for Elementary Cloud. The documentation doesn't explicitly mention this feature by name. However, I can share what I know about how Elementary Cloud works with tests and monitoring: According to the comparison between OSS and Cloud implementations: 1. Pipeline Health Monitors: In Elementary Cloud, these "Run in Cloud" rather than in dbt: - "Tests execution: Run in dbt [OSS] vs Run in Cloud [Cloud]" - "Time to detection: Only when dbt runs [OSS] vs As soon as the problem happens, including sources [Cloud]" [Data-tests: Anomaly Detection Tests - OSS vs Cloud](https://docs.elementary-data.com/data-tests/anomaly-detection-tests-oss-vs-cloud) 2. Data Quality Metrics: For these, Elementary Cloud uses a hybrid approach: - "Tests execution: Run in dbt [OSS] vs Metrics collection in dbt, detection in Cloud [Cloud]" [Data-tests: Anomaly Detection Tests - OSS vs Cloud](https://docs.elementary-data.com/data-tests/anomaly-detection-tests-oss-vs-cloud) This suggests that Elementary Cloud can run certain monitoring processes and anomaly detection in the cloud platform itself, rather than requiring them to be executed as part of your dbt runs. If you're looking for more specific information about a "Runs in cloud" feature, I'd recommend: 1. Checking the latest Elementary Cloud documentation 2. Reaching out to Elementary support through their community Slack 3. Scheduling a demo with their team at [Book a call with our team](https://meetings-eu1.hubspot.com/joost-boonzajer-flaes/intro-call-docs)

Based on the knowledge sources, Alert actions in Elementary Cloud are features that allow users to interact with and manage incidents directly from the alert notification itself. Here's what they include: ## Alert Actions in Elementary Cloud Alert actions enable you to: 1. Change incident status and assignee: You can update the status of an incident (Open, Acknowledged, Resolved) and assign it to team members directly from the alert notification [Incident Management](https://docs.elementary-data.com/features/alerts-and-incidents/incident-management). 2. Collaborate on incident resolution: Alert actions help teams collaborate more effectively by making it clear who is working on what and what the current status is [Status and assignee for alerts](https://docs.elementary-data.com/changelog#status-and-assignee-for-alerts). 3. Manage incidents from messaging apps: For example, in Slack integration, "The alerts include rich context, and you can change the incident status and asssigne from the alert itself" [Slack Integration](https://docs.elementary-data.com/cloud/integrations/alerts/slack). ## Recent Improvements In April 2024, Elementary added status and assignee selection to their alerts: "An alert is just the start of a triage and response process. We have big plans for making this process much more effective. The first step was alert rules, and now we are introducing status and assignee selection in our alerts. This should help you manage incidents and collaborate more effectively with your team." [Status and assignee for alerts](https://docs.elementary-data.com/changelog#status-and-assignee-for-alerts) ## Integration with Incident Management These alert actions are part of Elementary's broader incident management capabilities, which include: - Automatic grouping of related failures into a single incident - Tracking incident status, severity, and assignee - Centralized incident management interface - Automated resolution when issues are fixed This approach addresses the need for more actionable alerts that one community member expressed: "How am I going to make sure an alert is addressed, marked as complete, integrated with runbooks/guides, and routed to the right person to fix it without losing visibility across all tables/runs of errors?" [Slack forum question](https://slack.com/archives/C02CTC89LAX/p1695735775650729) Would you like more information about how alert actions work with specific integrations like Slack, MS Teams, or incident management platforms?

Yes, you can disable Elementary hooks locally to improve performance during development. This is a common need since the hooks can slow down local development cycles. ## Options to disable hooks locally ### Option 1: Disable specific hooks using vars Add these variables to your

dbt_project.yml

file under the

vars

section:

yaml vars:   disable_run_results: "{{ target.name != 'prod' }}"   disable_tests_results: "{{ target.name != 'prod' }}"   disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"   disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"

This disables the hooks based on your target environment [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1677157737267389). ### Option 2: Disable via command line For one-time disabling, you can add variables directly to your dbt command:

--vars '{"disable_dbt_artifacts_autoupload": true}'

This can be added to any dbt command [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1711555630207219). ### Option 3: Disable the entire package If you don't need any Elementary functionality locally, you can disable the entire package:

yaml models:   elementary:     +enabled: "{{ target.name in ['prod','analytics'] }}"

However, be careful with this approach if you have models that reference Elementary tables, as they will fail to compile [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1712570910953069). ## Important notes 1. Even with the hooks disabled, the hook will still run but won't do anything, which is expected behavior [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1669625539486519). 2. If you completely disable the Elementary package and have models that reference Elementary tables, you'll get compilation errors [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1699907896171609). 3. Recent versions of Elementary (v0.6.1+) include performance improvements to the hooks with a caching mechanism, which might help if disabling isn't an option [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1674814915518089). Would you like more specific guidance for your particular setup?

I understand you're having an issue where Elementary Cloud is not being updated after configuring hooks to only run in your production environment. Based on the knowledge sources, you're missing one important variable in your configuration. You need to add

disable_dbt_artifacts_autoupload

to your list of disabled hooks. This is crucial because it controls the upload of dbt metadata models that Elementary Cloud needs to stay updated. Here's the complete configuration you should use in your `dbt_project.yml`:

yaml vars:   disable_run_results: "{{ target.name != 'prod_Hex5674v_sec' }}"   disable_tests_results: "{{ target.name != 'prod_Hex5674v_sec' }}"   disable_dbt_artifacts_autoupload: "{{ target.name != 'prod_Hex5674v_sec' }}"   disable_dbt_invocation_autoupload: "{{ target.name != 'prod_Hex5674v_sec' }}"

As explained in the [Elementary documentation](https://docs.elementary-data.com/dbt/on-run-end_hooks#can-i-disable-the-on-run-end-hooks%3F), disabling

dbt_artifacts_autoupload

prevents the metadata models from being updated, which includes important information about your dbt models, tests, and sources. If you've disabled the artifacts autoupload and still want to keep Elementary Cloud updated, the documentation recommends: "If you disable the artifacts autoupload, we recommend your run

dbt run --select elementary.edr.dbt_artifacts

every time you deploy changes to your project." This way, you can keep your development environment clean while ensuring your production environment properly updates Elementary Cloud with the necessary metadata.

Based on the knowledge sources, Elementary doesn't only send Slack alerts for incidents. It can send alerts for various types of data issues, not just incidents. ## What Elementary Can Send Alerts For Elementary can send Slack alerts for: - Failures and/or warnings of dbt tests - Failures and/or warnings of Elementary tests (including anomaly detection) - Model runs failures - Source freshness failures - Custom SQL test failures As stated in the documentation: "Elementary can send alerts about: Failures and/or warnings of dbt tests, Failures and/or warnings Elementary tests, Model runs failures, Source freshness failures" [Elementary alerts](https://docs.elementary-data.com/oss/guides/alerts/elementary-alerts). ## Alerts vs. Incidents In Elementary Cloud, there's a distinction between alerts and incidents: - Alert: "Notification about an event that indicates a data issue." - Incident: "A data issue that starts with an event, but can include several events grouped to an incident. An incident has a start time, status, severity, assignee and end time." [Alerts and Incidents Overview](https://docs.elementary-data.com/features/alerts-and-incidents/alerts-and-incidents-overview) So alerts are the notifications themselves, while incidents are the tracked issues that may contain multiple related alerts. ## What's Not Included in Alerts Elementary does not send alerts for: - Successful model completions (unless you create a custom solution) - Seed errors (as confirmed by support staff: "elementary does not send alerts for errors in seeds" [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1706799802007729)) If you're only receiving incident alerts and not other types of alerts, you might need to check your configuration settings. For example, you might have disabled certain types of alerts with variables like:

yaml vars:   disable_model_alerts: true   disable_test_alerts: true   disable_warn_alerts: true

Would you like me to help troubleshoot why you might only be receiving incident alerts?

I can help you change the timestamp_column for freshness_anomalies tests in Elementary Data. Based on the knowledge sources, there are a few ways to configure this: ## Option 1: Configure at the test level You can specify the timestamp_column directly in your test configuration:

yaml models:   - name: your_model_name     tests:       - elementary.freshness_anomalies:           timestamp_column: your_timestamp_column_name

This is the most direct way to specify which column should be used for freshness anomaly detection [Data-tests > Anomaly-detection-tests](https://docs.elementary-data.com/data-tests/anomaly-detection-tests/freshness-anomalies). ## Option 2: Configure at the model level You can set the timestamp_column at the model level, which will be used by all Elementary tests that need a timestamp:

yaml models:   - name: your_model_name     config:       elementary:         timestamp_column: your_timestamp_column_name     tests:       - elementary.freshness_anomalies

This approach is useful when you have multiple Elementary tests that should all use the same timestamp column [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1681854412520199). ## Special cases for timestamp columns If your timestamp column is in a special format, there are solutions: 1. For string columns containing epoch time: You can use a SQL expression in your timestamp_column configuration. For example:

yaml    config:      elementary:        timestamp_column: TO_TIMESTAMP(inserted_at_epoch_string)

[Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1686748178330809) 2. For text columns that need casting: You can use casting in your timestamp_column configuration:

yaml    meta:      elementary:        timestamp_column: "timestamp::timestamp"

[Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1687385776940869) After making these changes, you'll need to run your dbt project to apply the configuration changes. Elementary Cloud will then use the updated timestamp column for freshness anomaly detection. Note that if you're using Elementary Cloud, these configurations will still be applied through your dbt project code, as Elementary Cloud uses a configuration-as-code approach.

@kapa.ai What about raw sources ?

I understand you're asking about how to configure timestamp columns for raw sources in Elementary Data, following our discussion about freshness_anomalies tests. Yes, you can definitely configure freshness monitoring for raw sources in Elementary. Here's how: ## Configuring timestamp columns for sources You can add Elementary tests to your sources in your schema.yml file, similar to how you would for models:

yaml version: 2 sources:  - name: source_name    schema: schema_name    loaded_at_field: "received_at"  # This is used for dbt source freshness    tables:      - name: raw_table_name        tests:          - elementary.freshness_anomalies:              timestamp_column: created_at  # Use your actual timestamp column              tags: ["elementary"]

As shown in a Slack forum answer, you can apply Elementary tests directly to source tables [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1670241799990399). ## For sources without timestamp columns If your raw source doesn't have a timestamp column, you have a few options: 1. Use a different test: Instead of freshness_anomalies, you might want to use schema_changes or volume_anomalies tests which don't require a timestamp column. 2. Create a generic test: As suggested in a forum answer, "you can create a generic test which gets a timestamp_column and error_after_days params and query your model with a where filter that checks if there is data after the (timestamp_column - errors_after_days)" [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1684401442974139). ## For sources with non-standard timestamp formats If your raw source has a timestamp in a non-standard format (like text):

yaml sources:   - name: source_name     tables:       - name: raw_table_name         tests:           - elementary.freshness_anomalies:               timestamp_column: "timestamp::timestamp"  # Cast to timestamp

This approach was discussed in a forum where a user needed to cast a text column to timestamp [Slack forum answer](https://slack.com/archives/C02CTC89LAX/p1687385776940869). Would you like more specific guidance for your particular raw source setup?