https://linen.dev logo
Join Slack
Powered by
<@U092UM6212T> i'm currently evaluation Elementar...
# ask-elementary-ai
n
@Stefan Fuchs i'm currently evaluation Elementary OSS Currently i try to run it from our Airflow pipeline, but i get a permission error, because elementary tries to write to some folder in the python env (see error message below). Should it work out of the box? Anything special to consider to run it in Airflow? I'm using a custom Operator that inherits from BashOperator to run the cli tool
Copy code
...
[2025-06-27, 08:24:38 UTC] {subprocess.py:88} INFO - Running command: ['/usr/bin/bash', '-c', '# Set up Elementary environment\nexport HOME=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic\nexport DBT_PROFILES_DIR=/tmp\nexport DBT_LOG_PATH=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/logs\nexport DBT_PACKAGES_DIR=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/dbt_packages\n\n# Create necessary directories\nmkdir -p $DBT_LOG_PATH\nmkdir -p $DBT_PACKAGES_DIR\n\n# Log environment for debugging\necho "Elementary Monitor Operator - Environment Setup:"\necho "HOME: $HOME"\necho "DBT_PROFILES_DIR: $DBT_PROFILES_DIR"\necho "DBT_LOG_PATH: $DBT_LOG_PATH"\necho "DBT_PACKAGES_DIR: $DBT_PACKAGES_DIR"\necho "Working Directory: $(pwd)"\n\n# Execute Elementary command\nedr monitor --profiles-dir /tmp --target-path /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/edr_output/elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00 --project-dir /usr/local/airflow/dbt/avalanche -t dev --env dev -d 7 --config-dir /tmp/elementary_config_6ou20bfb\nELEMENTARY_EXIT_CODE=$?\n\n# Log command result\nif [ $ELEMENTARY_EXIT_CODE -eq 0 ]; then\n    echo "Elementary monitor command completed successfully"\nelse\n    echo "Elementary monitor command failed with exit code $ELEMENTARY_EXIT_CODE"\nfi\n\n# Clean up temporary files\nrm -f /tmp/tmpn3_4qx78_profiles.yml\nrm -rf /tmp/elementary_config_6ou20bfb\n\nexit $ELEMENTARY_EXIT_CODE']
[2025-06-27, 08:24:39 UTC] {subprocess.py:99} INFO - Output:
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - Elementary Monitor Operator - Environment Setup:
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - HOME: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_PROFILES_DIR: /tmp
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_LOG_PATH: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/logs
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_PACKAGES_DIR: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/dbt_packages
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - Working Directory: /tmp/airflowtmpd5gw65t5
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -     ________                          __
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -    / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -                                                  /____/
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Running with edr=0.19.0
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.cli.cli:Running with edr=0.19.0
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Installing packages for edr internal dbt package...
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.clients.dbt.command_line_dbt_runner:Installing packages for edr internal dbt package...
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Running dbt command --log-format json deps --project-dir /usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp --target dev
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json deps --project-dir /usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp --target dev
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - {"data": {"exc": "[Errno 13] Permission denied: 'dbt_packages'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "6cf58c08-32ed-47a6-b665-7ce0caa3f391", "level": "error", "msg": "Encountered an error:\n[Errno 13] Permission denied: 'dbt_packages'", "name": "MainEncounteredError", "pid": 290, "thread": "MainThread", "ts": "2025-06-27T08:24:42.174015Z"}}
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - {"data": {"stack_trace": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 153, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 103, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 218, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 264, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/main.py\", line 455, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/task/deps.py\", line 229, in run\n    system.make_directory(self.project.packages_install_path)\n  File \"/usr/local/lib/python3.12/functools.py\", line 912, in wrapper\n    return dispatch(args[0].__class__)(*args, **kw)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 209, in _\n    raise e\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 203, in _\n    os.makedirs(path)\n  File \"<frozen os>\", line 225, in makedirs\nPermissionError: [Errno 13] Permission denied: 'dbt_packages'\n"}, "info": {"category": "", "code": "Z003", "extra": {}, "invocation_id": "6cf58c08-32ed-47a6-b665-7ce0caa3f391", "level": "error", "msg": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 153, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 103, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 218, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 264, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/main.py\", line 455, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/task/deps.py\", line 229, in run\n    system.make_directory(self.project.packages_install_path)\n  File \"/usr/local/lib/python3.12/functools.py\", line 912, in wrapper\n    return dispatch(args[0].__class__)(*args, **kw)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 209, in _\n    raise e\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 203, in _\n    os.makedirs(path)\n  File \"<frozen os>\", line 225, in makedirs\nPermissionError: [Errno 13] Permission denied: 'dbt_packages'\n", "name": "MainStackTrace", "pid": 290, "thread": "MainThread", "ts": "2025-06-27T08:24:42.177600Z"}}
...
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO -     raise DbtCommandError(
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - [Errno 13] Permission denied: 'dbt_packages'
[2025-06-27, 08:24:43 UTC] {subprocess.py:106} INFO - Elementary monitor command failed with exit code 1
[2025-06-27, 08:24:43 UTC] {subprocess.py:110} INFO - Command exited with return code 1
[2025-06-27, 08:24:43 UTC] {taskinstance.py:3313} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 768, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 734, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 424, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/plugins/operators/ElementaryMonitorOperator.py", line 126, in execute
    return super().execute(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 424, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/operators/bash.py", line 276, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
...
k
A message from kapa.ai
You're encountering a common issue when running Elementary OSS in Airflow: permission errors when the CLI tries to write to the internal dbt project's dbt_packages directory inside the Python environment. This is a known challenge, especially in managed or containerized environments where the Python site-packages directory is often read-only. Why does this happen? Elementary uses an internal dbt project (bundled with the Python package) and, on execution, runs 
dbt deps
to install or update its dependencies. By default, this tries to write to a 
dbt_packages
directory inside the installed package path (e.g., 
/usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project/dbt_packages
). If this location is not writable by the user running the Airflow task, you get a 
[Errno 13] Permission denied: 'dbt_packages'
error [see example](https://slack.com/archives/C02CTC89LAX/p1681911667304009). Should it work out of the box? Not always. In many Airflow or containerized setups, the Python environment is read-only for non-root users, so you need to take extra steps to ensure Elementary can write where it needs to. What can you do? 1. Set a Writable DBT Packages Directory You can control where Elementary installs dbt dependencies by setting the 
DBT_PACKAGES_FOLDER
environment variable to a writable location (e.g., 
/tmp/edr_dbt_packages
). This is an internal variable not always documented, but it is used by Elementary to override the default path [see this discussion](https://slack.com/archives/C02CTC89LAX/p1716798424976069):
Copy code
bash    export DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages
Add this to your BashOperator environment or before running the 
edr
command. 2. Change Permissions (if possible) If you control the environment, you can make the internal dbt project directory writable before running Elementary:
Copy code
bash    sudo chmod -R 777 /usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project
However, this is often not possible in managed environments or due to security policies [see here](https://slack.com/archives/C02CTC89LAX/p1696246081105209). 3. Install Elementary in a Writable Directory If you use 
pip install --target
to install Elementary into a directory you control (e.g., 
/tmp/elementary
), and ensure your Airflow task runs from there, you can avoid the read-only site-packages issue [see example](https://slack.com/archives/C02CTC89LAX/p1686278697336859). 4. Container Build Step If you build your own Docker image, you can pre-run 
dbt deps
in the internal dbt project directory as part of the image build, ensuring dependencies are already present and writable [see example](https://slack.com/archives/C02CTC89LAX/p1686278697336859). Summary: Elementary OSS does not always work out of the box in Airflow due to Python environment permissions. The most robust solution is to set the 
DBT_PACKAGES_FOLDER
environment variable to a writable directory before running Elementary. This approach is widely used and has resolved similar issues for other users. If you continue to have issues, please share your updated environment variables and command setup for further troubleshooting.
s
i called it 
DBT_PACKAGES_DIR
🤦
😅 1
32 Views
Open in Slack
PreviousNext
Powered by