:hello: i'm currently evaluation Elementary OSS Cu...
# community-support
s
hello i'm currently evaluation Elementary OSS Currently i try to run it from our Airflow pipeline, but i get a permission error, because elementary tries to write to some folder in the python env (see error message below). Should it work out of the box? Anything special to consider to run it in Airflow? I'm using a custom Operator that inherits from BashOperator to run the cli tool
Copy code
...
[2025-06-27, 08:24:38 UTC] {subprocess.py:88} INFO - Running command: ['/usr/bin/bash', '-c', '# Set up Elementary environment\nexport HOME=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic\nexport DBT_PROFILES_DIR=/tmp\nexport DBT_LOG_PATH=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/logs\nexport DBT_PACKAGES_DIR=/tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/dbt_packages\n\n# Create necessary directories\nmkdir -p $DBT_LOG_PATH\nmkdir -p $DBT_PACKAGES_DIR\n\n# Log environment for debugging\necho "Elementary Monitor Operator - Environment Setup:"\necho "HOME: $HOME"\necho "DBT_PROFILES_DIR: $DBT_PROFILES_DIR"\necho "DBT_LOG_PATH: $DBT_LOG_PATH"\necho "DBT_PACKAGES_DIR: $DBT_PACKAGES_DIR"\necho "Working Directory: $(pwd)"\n\n# Execute Elementary command\nedr monitor --profiles-dir /tmp --target-path /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/edr_output/elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00 --project-dir /usr/local/airflow/dbt/avalanche -t dev --env dev -d 7 --config-dir /tmp/elementary_config_6ou20bfb\nELEMENTARY_EXIT_CODE=$?\n\n# Log command result\nif [ $ELEMENTARY_EXIT_CODE -eq 0 ]; then\n    echo "Elementary monitor command completed successfully"\nelse\n    echo "Elementary monitor command failed with exit code $ELEMENTARY_EXIT_CODE"\nfi\n\n# Clean up temporary files\nrm -f /tmp/tmpn3_4qx78_profiles.yml\nrm -rf /tmp/elementary_config_6ou20bfb\n\nexit $ELEMENTARY_EXIT_CODE']
[2025-06-27, 08:24:39 UTC] {subprocess.py:99} INFO - Output:
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - Elementary Monitor Operator - Environment Setup:
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - HOME: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_PROFILES_DIR: /tmp
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_LOG_PATH: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/logs
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - DBT_PACKAGES_DIR: /tmp/elementary_elementary_operators_test_test_elementary_monitor_alerts_manual__2025-06-27T08:24:31.845126+00:00_6w25h0ic/dbt_packages
[2025-06-27, 08:24:39 UTC] {subprocess.py:106} INFO - Working Directory: /tmp/airflowtmpd5gw65t5
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -     ________                          __
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -    / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO -                                                  /____/
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - Any feedback and suggestions are welcomed! join our community here - <https://bit.ly/slack-elementary>
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Running with edr=0.19.0
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.cli.cli:Running with edr=0.19.0
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Installing packages for edr internal dbt package...
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.clients.dbt.command_line_dbt_runner:Installing packages for edr internal dbt package...
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - 2025-06-27 08:24:41 — INFO — Running dbt command --log-format json deps --project-dir /usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp --target dev
[2025-06-27, 08:24:41 UTC] {subprocess.py:106} INFO - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json deps --project-dir /usr/local/lib/python3.12/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp --target dev
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - {"data": {"exc": "[Errno 13] Permission denied: 'dbt_packages'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "6cf58c08-32ed-47a6-b665-7ce0caa3f391", "level": "error", "msg": "Encountered an error:\n[Errno 13] Permission denied: 'dbt_packages'", "name": "MainEncounteredError", "pid": 290, "thread": "MainThread", "ts": "2025-06-27T08:24:42.174015Z"}}
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - {"data": {"stack_trace": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 153, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 103, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 218, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 264, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/main.py\", line 455, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/task/deps.py\", line 229, in run\n    system.make_directory(self.project.packages_install_path)\n  File \"/usr/local/lib/python3.12/functools.py\", line 912, in wrapper\n    return dispatch(args[0].__class__)(*args, **kw)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 209, in _\n    raise e\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 203, in _\n    os.makedirs(path)\n  File \"<frozen os>\", line 225, in makedirs\nPermissionError: [Errno 13] Permission denied: 'dbt_packages'\n"}, "info": {"category": "", "code": "Z003", "extra": {}, "invocation_id": "6cf58c08-32ed-47a6-b665-7ce0caa3f391", "level": "error", "msg": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 153, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 103, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 218, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/requires.py\", line 264, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/cli/main.py\", line 455, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt/task/deps.py\", line 229, in run\n    system.make_directory(self.project.packages_install_path)\n  File \"/usr/local/lib/python3.12/functools.py\", line 912, in wrapper\n    return dispatch(args[0].__class__)(*args, **kw)\n           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 209, in _\n    raise e\n  File \"/usr/local/lib/python3.12/site-packages/dbt_common/clients/system.py\", line 203, in _\n    os.makedirs(path)\n  File \"<frozen os>\", line 225, in makedirs\nPermissionError: [Errno 13] Permission denied: 'dbt_packages'\n", "name": "MainStackTrace", "pid": 290, "thread": "MainThread", "ts": "2025-06-27T08:24:42.177600Z"}}
...
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO -     raise DbtCommandError(
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
[2025-06-27, 08:24:42 UTC] {subprocess.py:106} INFO - [Errno 13] Permission denied: 'dbt_packages'
[2025-06-27, 08:24:43 UTC] {subprocess.py:106} INFO - Elementary monitor command failed with exit code 1
[2025-06-27, 08:24:43 UTC] {subprocess.py:110} INFO - Command exited with return code 1
[2025-06-27, 08:24:43 UTC] {taskinstance.py:3313} ERROR - Task failed with exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 768, in _execute_task
    result = _execute_callable(context=context, **execute_callable_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 734, in _execute_callable
    return ExecutionCallableRunner(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
    return self.func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 424, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/airflow/plugins/operators/ElementaryMonitorOperator.py", line 126, in execute
    return super().execute(context)
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 424, in wrapper
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/airflow/operators/bash.py", line 276, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
...
n
Hey @Stefan Fuchs! Welcome to Elementary drop stoked I've forwarded your question to #C08HW57FGN9, please let us know if you need any further help!
thankyou 1