https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> dbt_project.yml is a single shared ...
# ask-elementary-ai
s
@kapa.ai dbt_project.yml is a single shared file used by all environments. Any variable set there applies everywhere. By moving enable_elementary to the .env files, we get per-environment control ā€” since each environment has its own .env file.
k
You're correct in your reasoning. Since 
dbt_project.yml
is shared across all environments, any value set directly there applies globally. Using environment variables (via 
.env
files or your CI/CD system's environment variable management) gives you per-environment control. Here's how this pattern works with Elementary: 1. Use 
env_var()
in 
dbt_project.yml
Copy code
models:
  elementary:
    +schema: elementary
    +enabled: "{{ env_var('ENABLE_ELEMENTARY', 'false') == 'true' }}"

vars:
  disable_run_results: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_tests_results: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_dbt_artifacts_autoupload: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_dbt_invocation_autoupload: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
2. Set 
ENABLE_ELEMENTARY
per environment in your 
.env
files ā€¢ Local dev `.env`: 
ENABLE_ELEMENTARY=false
ā€¢ Staging/prod `.env`: 
ENABLE_ELEMENTARY=true
Important note: The 
+enabled
flag alone is not sufficient ā€” it controls the Elementary models, but the on-run-end hooks are controlled separately via the 
disable_*
vars. You need both to fully disable Elementary. [disable env var] Also note that if you use 
+enabled: false
, Elementary tests will also be disabled, not just the artifact uploads. If you want Elementary tests to still run but just skip artifact collection, the hook disablement vars approach is preferred over disabling the entire package. [dev-prod config] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai how can I toggle Elementary on/off per env ?
k
Based on the conversation so far, here's a summary of the recommended approaches to toggle Elementary on/off per environment: Option 1: Hook disablement vars (Recommended) This keeps Elementary models and tests available but skips artifact collection in non-prod environments:
Copy code
vars:
  disable_run_results: "{{ target.name != 'prod' }}"
  disable_tests_results: "{{ target.name != 'prod' }}"
  disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"
  disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
[OSS FAQ; reduce on-run-end time] Option 2: Disable the entire package per target
Copy code
models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.name == 'prod' }}"
āš ļø This also disables Elementary tests, not just artifact uploads. [dev-prod config] Option 3: Use an environment variable (for .env-based control)
Copy code
models:
  elementary:
    +schema: elementary
    +enabled: "{{ env_var('ENABLE_ELEMENTARY', 'false') == 'true' }}"

vars:
  disable_run_results: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_tests_results: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_dbt_artifacts_autoupload: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
  disable_dbt_invocation_autoupload: "{{ env_var('ENABLE_ELEMENTARY', 'false') != 'true' }}"
Then set 
ENABLE_ELEMENTARY=true
or 
false
in each environment's 
.env
file. [disable env var] Key point: The 
+enabled
flag alone is not enough ā€” you must also set the 
disable_*
vars to fully suppress the on-run-end hooks. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai run dbt elementary init running for 25 min
k
Long initial runs are a known and expected behavior with Elementary. Here's what's happening and how to address it: Why the First Run Takes So Long The initial 
dbt run --select elementary
(or the first run after installing Elementary) can take a very long time because: 1. No metadata cache exists yet ā€” Elementary needs to upload metadata for all resources in your project (models, tests, sources, etc.) from scratch. For large projects, this can take many minutes or even over an hour. [on-run-end hooks] 2. 
dbt_columns
is especially heavy ā€” This model scans column metadata across your entire project and is frequently the biggest bottleneck. One user with ~90K rows reported it taking 30+ minutes. [on-run-end hooks] 3. 
dbt_tests
can be slow ā€” If you have many tests (e.g., 6000+), the 
dbt_tests
model can take several minutes on its own. [Slack thread] > Good news: Subsequent runs should be much faster thanks to Elementary's metadata cache (available for dbt 1.4.0+), which only re-uploads what has changed. [on-run-end hooks] What You Can Do Right Now 1. Disable column upload (biggest win):
Copy code
vars:
  columns_upload_strategy: none
Or the older flag:
Copy code
vars:
  disable_dbt_columns_autoupload: true
One user went from ~30 minutes to 4ā€“5 minutes by disabling this. [Slack thread] 2. Disable artifact autoupload and run manually only on deploy:
Copy code
vars:
  disable_dbt_artifacts_autoupload: true
Then run manually when needed:
Copy code
dbt run --select edr.dbt_artifacts
[on-run-end hooks] 3. Make sure you're on dbt 1.4.0+ ā€” The metadata cache that prevents full re-uploads on every run was introduced in dbt 1.4.0. If you're on an older version, every run will be slow. [on-run-end hooks] If the problem persists after the first run, sharing your 
dbt.log
with the Elementary team can help identify the specific bottleneck. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai how to get slack alerts
k
Here's how to set up Slack alerts with Elementary OSS: 1. Create a Slack App Go to the Slack app creation page, create a new app from scratch, and connect it to your workspace. [Slack setup] 2. Choose: Token or Webhook Token is recommended if you need multiple channels or report sharing. [Slack setup] Token setup Add these Bot Token Scopes under "OAuth & Permissions": 
channels:join
, 
channels:read
, 
chat:write
, 
files:write
, 
users:read
, 
users:read.email
, 
groups:read
. Then install the app to your workspace to generate a token. Webhook setup Enable "Incoming Webhooks" in your app, add a new webhook to your desired channel, and copy the URL. 3. Configure Elementary Via CLI:
Copy code
# Token
edr monitor --slack-token <your_slack_token> --slack-channel-name <channel_name>

# Webhook
edr monitor --slack-webhook <your_slack_webhook_url>
Via `~/.edr/config.yml`:
Copy code
# Token
slack:
  token: <your_slack_token>
  channel_name: <slack_channel_to_post_at>
  group_alerts_by: table  # or alert

# Webhook
slack:
  notification_webhook: <your_slack_webhook_url>
  group_alerts_by: table  # or alert
[Send Slack alerts] 4. Run After Your dbt Tests
Copy code
dbt run
dbt test
edr monitor
Elementary will send alerts for test failures, warnings, model run failures, and source freshness failures. [Setup Slack Alerts] > Note: The 
--slack-channel-name
is the default channel. You can override it per model or test using 
alerts_config
in your 
.yml
files. [custom channels] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
s
@kapa.ai where to install the CLI / edr ?
k
You can install the Elementary CLI (
edr
) using 
pip
. Here's how: Installation 1. Install the base package:
Copy code
pip install elementary-data
2. Install the connector for your data warehouse (run only the one you need):
Copy code
pip install 'elementary-data[snowflake]'
pip install 'elementary-data[bigquery]'
pip install 'elementary-data[redshift]'
pip install 'elementary-data[databricks]'
pip install 'elementary-data[athena]'
pip install 'elementary-data[trino]'
pip install 'elementary-data[clickhouse]'
# Postgres doesn't require this step
[CLI install] 3. Validate the installation:
Copy code
edr --help
If you see the help output, the installation was successful. If you get 
command not found: edr
, check the troubleshooting guide. [Quickstart CLI] Tips ā€¢ Python version: Elementary supports Python 3.9ā€“3.12. [Quickstart CLI] ā€¢ Anaconda users: Create a dedicated conda environment first to avoid dependency conflicts:conda create -n elementary-env python=3.9 conda activate elementary-env pip install 'elementary-data[snowflake]' # or your platform ā€¢ *Install from source (alternative):*git clone https://github.com/elementary-data/elementary pip install ./elementary[CLI install] After Installation You'll need to configure a 
profiles.yml
file (at 
~/.dbt/profiles.yml
) with an 
elementary
profile pointing to your data warehouse. Generate a template with:
Copy code
dbt run-operation elementary.generate_elementary_cli_profile
[CLI install] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai in a workflow how would this be implemented?
k
Based on the conversation history, you're asking how to implement the Elementary CLI (
edr
) in a CI/CD workflow. Here are the two most common approaches: Option 1: GitHub Actions Use the dedicated `elementary-data/run-elementary-action`:
Copy code
name: Run Elementary
on:
  schedule:
    - cron: '0 9,13,19 * * 1-5'
  workflow_dispatch:

jobs:
  elementary:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout dbt project
        uses: actions/checkout@v3
      - name: Run Elementary
        uses: elementary-data/run-elementary-action@v1.12
        with:
          warehouse-type: snowflake
          profiles-yml: ${{ secrets.ELEMENTARY_PROFILES_YML }}
          edr-command: |
            edr monitor --slack-token "${{ secrets.SLACK_TOKEN }}" --slack-channel-name "${{ secrets.SLACK_CHANNEL_NAME }}"
            &&
            edr send-report --s3-bucket-name "${{ secrets.S3_BUCKET_NAME }}"
      - name: Upload report
        uses: actions/upload-artifact@v3
        with:
          name: report.html
          path: report.html
      - name: Upload log
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: edr.log
          path: edr.log
[GitHub Actions docs] Key requirements: ā€¢ Store your 
profiles.yml
content as a GitHub secret (e.g., 
ELEMENTARY_PROFILES_YML
) ā€¢ Specify your warehouse type ā€¢ dbt must run before Elementary ā€” add it as a prior step if needed Option 2: Airflow DAG If you're orchestrating dbt Core with Airflow, run 
edr monitor
as a 
BashOperator
task after `dbt test`:
Copy code
from datetime import datetime
import os
from airflow import DAG
from airflow.operators.bash_operator import BashOperator

with DAG(
    dag_id='dbt_edr_dag',
    start_date=datetime(2022, 9, 1),
    schedule_interval='0 0 * * *',
) as dag:

    dbt_run = BashOperator(
        task_id='dbt_run',
        bash_command=f"dbt run --profiles-dir {os.getenv('DBT_PROFILES_DIR')}"
    )

    dbt_test = BashOperator(
        task_id='dbt_test',
        bash_command=f"dbt test --profiles-dir {os.getenv('DBT_PROFILES_DIR')}"
    )

    edr_monitor = BashOperator(
        task_id='elementary_monitor',
        bash_command=f"edr monitor --profiles-dir {os.getenv('DBT_PROFILES_DIR')} --slack-token {os.getenv('SLACK_TOKEN')} --slack-channel-name {os.getenv('EDR_SLACK_CHANNEL')}"
    )

    dbt_run >> dbt_test >> edr_monitor
[Airflow scheduling] Key requirement: 
edr
must be installed in the Airflow environment. Option 3: GitLab CI
Copy code
image:
  name: ghcr.io/elementary-data/elementary:<TAG>
  entrypoint: [""]

elementary:
  script:
    - echo "$PROFILES_YML" > profiles.yml
    - edr monitor --slack-webhook "$SLACK_WEBHOOK"
    - edr send-report --aws-access-key-id "$AWS_ACCESS_KEY_ID" --aws-secret-access-key "$AWS_SECRET_ACCESS_KEY" --s3-bucket-name "$S3_BUCKET_NAME"
  artifacts:
    paths:
      - edr.log
[GitLab CI] In all cases, the pattern is the same: run dbt first, then run 
edr
. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai how to add edr in the airflow environment
k
To add 
edr
to your Airflow environment, you need to install the 
elementary-data
Python package in the same Python environment that Airflow uses to execute tasks. Installation Run the following in your Airflow environment:
Copy code
pip install elementary-data
Add the connector for your data warehouse:
Copy code
pip install 'elementary-data[snowflake]'   # or bigquery, redshift, databricks, etc.
[CLI install] Key Requirements ā€¢ 
edr
must be installed in the Airflow worker environment ā€” not just on your local machine. [Airflow integration] ā€¢ A valid 
profiles.yml
with an 
elementary
profile must be accessible to the Airflow task. [CLI install] Common Issues ā€¢ 
edr
not found: If you get a "command not found" error, it means 
elementary-data
is not installed in the environment Airflow is using to run the task. You can add 
pip install elementary-data[<your-warehouse>]
as a step before your 
edr
commands in the DAG. [Slack thread] ā€¢ Permission errors writing `dbt_packages`: Elementary may try to write to a read-only directory. Fix this by setting: [Airflow integration]export DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages ā€¢ Docker-based Airflow: If you can't install packages directly, use the Elementary Docker image or the GitHub Action as alternatives. [Slack thread] Once Installed Use a 
BashOperator
to run 
edr
after your dbt tasks:
Copy code
edr_monitor = BashOperator(
    task_id='elementary_monitor',
    bash_command=f"edr monitor --profiles-dir {os.getenv('DBT_PROFILES_DIR')} --slack-token {os.getenv('SLACK_TOKEN')} --slack-channel-name {os.getenv('EDR_SLACK_CHANNEL')}"
)

dbt_run >> dbt_test >> edr_monitor
[Airflow scheduling] šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
30 Views
Open in Slack
PreviousNext
Powered by