https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> elementary and airflow
# ask-elementary-ai
s
@kapa.ai elementary and airflow
k
Here's an overview of how Elementary integrates with Apache Airflow: Running Elementary in Airflow The most common approach is to use a BashOperator in your Airflow DAG to run 
edr
CLI commands after your dbt runs and tests: [Airflow scheduling]
Copy code
from datetime import datetime
import os
from airflow import DAG
from airflow.operators.bash_operator import BashOperator

with DAG(
    dag_id='dbt_edr_dag',
    start_date=datetime(2022, 9, 1),
    schedule_interval='0 0 * * *',
) as dag:

    dbt_run = BashOperator(
        task_id='dbt_run',
        bash_command=f"dbt run --profiles-dir {os.getenv('DBT_PROFILES_DIR')}"
    )

    dbt_test = BashOperator(
        task_id='dbt_test',
        bash_command=f"dbt test --profiles-dir {os.getenv('DBT_PROFILES_DIR')}"
    )

    edr_monitor = BashOperator(
        task_id='elementary_monitor',
        bash_command=f"edr monitor --profiles-dir {os.getenv('DBT_PROFILES_DIR')} --slack-token {os.getenv('SLACK_TOKEN')} --slack-channel-name {os.getenv('EDR_SLACK_CHANNEL')}"
    )

    dbt_run >> dbt_test >> edr_monitor
Key Considerations Prerequisites • The Airflow environment must have 
edr
installed (it installs 
dbt
automatically). • A valid 
profiles.yml
must be accessible to the Airflow task. Credentials can be injected via environment variables. [dbt Cloud setup] Permission Errors A common issue is Elementary trying to write 
dbt_packages
to a read-only Python environment directory. The recommended fix is to set the 
DBT_PACKAGES_FOLDER
environment variable to a writable location: [permission fix]
Copy code
export DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages
Memory / OOM Issues (SIGKILL) If your Airflow task is killed with 
Signals.SIGKILL: 9
, it's likely an out-of-memory issue. Recommended mitigations: [memory reduction] • Reduce the data window: 
--days-back 1
• Lower execution limits: 
--executions-limit 120
• Increase Airflow worker memory allocation Concurrency with Cosmos If you use astronomer-cosmos (which parallelizes dbt models as individual Airflow tasks), concurrent writes to Elementary tables can cause transaction conflicts. The recommended approach is: [Cosmos concurrency] 1. Disable auto-upload in `dbt_project.yml`:
Copy code
vars:
     disable_dbt_artifacts_autoupload: true
2. Run 
dbt run -s elementary
in a single, dedicated serial job after your parallel model runs. Cloud Integration For Elementary Cloud, you can collect Airflow job metadata (job run URL, job name, job ID) visible in the lineage node info tab, and configure webhook-triggered syncs for near-real-time results. [Cloud Airflow] Alternative Deployment If installing 
edr
directly in Airflow is not feasible, Elementary provides a Docker image and a GitHub Action as alternatives. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai Elementary GitHub Actions vs Airflow Dags
k
Here's a comparison of the two approaches for running Elementary in production: GitHub Actions vs. Airflow DAGs for Elementary GitHub Actions Elementary provides a dedicated GitHub Action (
elementary-data/run-elementary-action
) that simplifies setup. [GitHub Actions docs] Typical use cases: • Triggering Elementary after a push/PR to your main branch • Running on a schedule (cron) • Sending reports to Slack or S3/GCS after dbt Cloud jobs complete (since dbt Cloud can't run 
edr
directly) Example workflow:
Copy code
name: Run Elementary
on:
  schedule:
    - cron: '0 9,13,19 * * 1-5'
  workflow_dispatch:

jobs:
  elementary:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout dbt project
        uses: actions/checkout@v3
      - name: Run Elementary
        uses: elementary-data/run-elementary-action@v1.12
        with:
          warehouse-type: snowflake
          profiles-yml: ${{ secrets.ELEMENTARY_PROFILES_YML }}
          edr-command: |
            edr monitor --slack-token "${{ secrets.SLACK_TOKEN }}" --slack-channel-name "${{ secrets.SLACK_CHANNEL_NAME }}"
            &&
            edr send-report --s3-bucket-name "${{ secrets.S3_BUCKET_NAME }}"
[GHA example] Key requirements: • 
profiles.yml
with an 
elementary
profile (stored as a GitHub secret) • Warehouse type specified • Note: dbt must run before Elementary — add it as a prior step if needed Potential pitfalls: • Organization restrictions may block external actions (workaround: fork the action into your org) • Keyfile paths should be absolute • Environment variables in 
profiles.yml
must be exposed in the workflow ------------------------- Airflow DAGs Best suited when you're already orchestrating dbt Core with Airflow and want Elementary to run as part of the same pipeline. [Airflow scheduling] Typical use case: • Running 
edr monitor
as a BashOperator task after 
dbt test
in the same DAG Example:
Copy code
edr_monitor = BashOperator(
    task_id='elementary_monitor',
    bash_command=f"edr monitor --profiles-dir {os.getenv('DBT_PROFILES_DIR')} --slack-token {os.getenv('SLACK_TOKEN')} --slack-channel-name {os.getenv('EDR_SLACK_CHANNEL')}"
)

dbt_run >> dbt_test >> edr_monitor
Key requirements: • 
edr
must be installed in the Airflow environment • 
profiles.yml
must be accessible to the worker ------------------------- Which to Choose? | Factor | GitHub Actions | Airflow DAGs | | - | - | - | | Already using Airflow for dbt | ❌ Redundant | ✅ Natural fit | | Using dbt Cloud | ✅ Best option | ⚠️ Possible but less common | | Simple setup | ✅ Dedicated action available | ⚠️ Requires 
edr
installed on workers | | Scheduling flexibility | ✅ Cron-based | ✅ Full DAG orchestration | The Elementary docs recommend orchestrating Elementary using the same system that orchestrates dbt — so if you're on dbt Core + Airflow, Airflow DAGs are the natural choice, while GitHub Actions is particularly useful for dbt Cloud users or simpler CI/CD-driven setups. [production deployment] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai two DAGs overlapping
k
When two Airflow DAGs (or any dbt jobs) run simultaneously and both trigger Elementary's 
on-run-end
hooks at the same time, you'll typically see transaction/concurrency errors like: • BigQuery: 
Transaction is aborted due to concurrent update against table ...dbt_models
[BQ concurrency] • Redshift: 
Database Error 1023
[Redshift concurrency] • Duplicate entries in metadata tables like 
dbt_tests
, 
dbt_models
, 
dbt_sources
[duplicate entries] • Duplicate Slack alerts [duplicate alerts] Recommended Solutions 1. Disable artifact autoupload (most common fix) Add this to your `dbt_project.yml`:
Copy code
vars:
  disable_dbt_artifacts_autoupload: true
Then run the artifacts update separately in a single, controlled job:
Copy code
dbt run --select elementary.edr.dbt_artifacts
This disables only the artifact upload part of the hook (item #3). Test results and invocation uploads are separate and use different variables. [disable autoupload] 2. Disable Elementary models from running automatically In 
dbt_project.yml
, disable Elementary models by default:
Copy code
models:
  your_project:
    +materialized: table
  elementary:
    +schema: elementary
    edr:
      +enabled: "{{ var('enable_elementary_models', false) }}"
Then update them explicitly in a single dedicated job:
Copy code
dbt run --select elementary --vars '{enable_elementary_models: true}'
[separate Elementary run] 3. Use Airflow Pools to serialize Elementary writes Create an Airflow pool with a single slot and assign it to any task that triggers Elementary hooks. This ensures only one DAG writes to Elementary tables at a time. [Airflow pool] 4. Run 
edr monitor
on its own schedule Instead of running 
edr monitor
at the end of every DAG, run it as a standalone job on a fixed interval (e.g., every 10 minutes). This avoids multiple concurrent 
edr monitor
executions. [standalone monitor] ------------------------- In summary, the most broadly recommended approach is option 1 (disable artifact autoupload) combined with a single serial job to refresh Elementary artifacts. This is explicitly documented in Elementary's troubleshooting guide. [troubleshooting guide] 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
31 Views
Open in Slack
PreviousNext
Powered by