https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> I have this error, when I try to la...
# ask-elementary-ai
d
@kapa.ai I have this error, when I try to launch edr from my dbt docker image.
Copy code
{"data": {"exc": "[Errno 13] Permission denied: '/usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project/package-lock.yml'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "aa32d24d-5362-4ad7-b2d4-b122b1285e7f", "level": "error", "msg": "Encountered an error:\n[Errno 13] Permission denied: '/usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project/package-lock.yml'", "name": "MainEncounteredError", "pid": 1, "thread": "MainThread", "ts": "2025-04-11T14:34:35.438654Z"}}
[2025-04-11, 14:34:35 UTC] {pod_manager.py:468} INFO - [base] {"data": {"stack_trace": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 138, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 101, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 201, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 247, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/main.py\", line 447, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/task/deps.py\", line 217, in run\n    self.lock()\n  File \"/usr/local/lib/python3.11/site-packages/dbt/task/deps.py\", line 204, in lock\n    with open(lock_filepath, \"w\") as lock_obj:\n         ^^^^^^^^^^^^^^^^^^^^^^^^\nPermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project/package-lock.yml'\n"}, "info": {"category": "", "code": "Z003", "extra": {}, "invocation_id": "aa32d24d-5362-4ad7-b2d4-b122b1285e7f", "level": "error", "msg": "Traceback (most recent call last):\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 138, in wrapper\n    result, success = func(*args, **kwargs)\n                      ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 101, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 201, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/requires.py\", line 247, in wrapper\n    return func(*args, **kwargs)\n           ^^^^^^^^^^^^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/cli/main.py\", line 447, in deps\n    results = task.run()\n              ^^^^^^^^^^\n  File \"/usr/local/lib/python3.11/site-packages/dbt/task/deps.py\", line 217, in run\n    self.lock()\n  File \"/usr/local/lib/python3.11/site-packages/dbt/task/deps.py\", line 204, in lock\n    with open(lock_filepath, \"w\") as lock_obj:\n         ^^^^^^^^^^^^^^^^^^^^^^^^\nPermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project/package-lock.yml'\n", "name": "MainStackTrace", "pid": 1, "thread": "MainThread", "ts": "2025-04-11T14:34:35.441892Z"}}
k
A message from kapa.ai
You're encountering a permission error when trying to run Elementary Data's CLI (edr) in your Docker container. This is a common issue that occurs because the Elementary CLI has an internal dbt project that needs to install dependencies, but the user running the command doesn't have write permissions to the necessary directories. ## Why this happens When you run Elementary CLI commands, it tries to execute 
dbt deps
on its internal dbt project located at 
/usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project
. The error shows that it's trying to write to a 
package-lock.yml
file in this directory but doesn't have permission. ## Solutions There are several ways to fix this issue: ### 1. Change permissions in your Dockerfile Add this command to your Dockerfile to give the appropriate user write permissions to the Elementary package directory:
Copy code
RUN chown -R your_user:your_group /usr/local/lib/python3.11/site-packages/elementary
Many users have solved this by adding similar commands to their Dockerfiles: [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1685408463792899) shows a user who fixed this with:
Copy code
RUN chown -R dbtuser /usr/local/lib/python3.8/site-packages/elementary
### 2. Run dbt deps for the internal project during Docker build Another approach is to pre-install the dependencies during the Docker build process:
Copy code
RUN cd $(python -c "import os; from elementary.monitor import dbt_project_utils; print(os.path.dirname(os.path.realpath(dbt_project_utils.__file__)))")/dbt_project && dbt deps
### 3. Use the DBT_PACKAGES_FOLDER environment variable As mentioned in a [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1716798424976069), you can control where Elementary installs its dbt dependencies by setting the 
DBT_PACKAGES_FOLDER
environment variable to a location where your user has write permissions:
Copy code
DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages
This environment variable isn't well-documented but was discovered by users examining the Elementary source code. ### 4. Use sudo (not recommended for production) For testing purposes only, you could run the command with sudo, but this isn't recommended for production environments. Choose the solution that best fits your environment and security requirements. The first option (changing permissions) is the most commonly used approach.
d
@kapa.ai It still doesn't work, and it's for production purpose. I don't want to use root for the container. I'm using a specific user.
k
A message from kapa.ai
I understand you're still facing issues with the permission error in your production environment, and you're correctly using a specific user rather than root for your container. For production environments, here are the most secure and recommended approaches to solve this permission issue without using root: ### 1. Set permissions during image build The most reliable solution is to modify your Dockerfile to set the correct permissions during build time:
Copy code
# After installing elementary-data RUN chown -R your_specific_user:your_specific_group /usr/local/lib/python3.11/site-packages/elementary
This approach has worked for many users in production environments as seen in the [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1674851653694939). ### 2. Use the DBT_PACKAGES_FOLDER environment variable This is an excellent production-ready solution that doesn't require changing permissions:
Copy code
# In your container environment configuration DBT_PACKAGES_FOLDER=/path/where/your/user/has/write/permissions
As mentioned in a [Slack discussion](https://slack.com/archives/C02CTC89LAX/p1716798424976069), this environment variable allows you to redirect where Elementary installs its dbt dependencies to a location where your specific user has write permissions. ### 3. Pre-install dependencies during build Another production-friendly approach is to run the dbt deps command during your Docker build:
Copy code
# In your Dockerfile RUN cd $(python -c "import os; from elementary.monitor import dbt_project_utils; print(os.path.dirname(os.path.realpath(dbt_project_utils.__file__)))")/dbt_project && dbt deps
This approach has been successful for users as documented in [Slack discussions](https://slack.com/archives/C02CTC89LAX/p1686278697336859). For production environments, I recommend trying the DBT_PACKAGES_FOLDER approach first as it's the least invasive and most secure option that doesn't require changing permissions or running as root.
d
@kapa.ai I'm using dbt image as a base, is that correct ?
k
A message from kapa.ai
Yes, using a dbt image as a base for your Elementary Data setup is a valid approach. Many users successfully run Elementary Data on top of dbt Docker images. When using a dbt base image, you'll still need to address the permission issues we discussed earlier. The error you're experiencing is specifically related to Elementary's internal dbt project trying to write to a location where your non-root user doesn't have permissions. For a production environment with a dbt base image, I recommend these approaches: 1. Set the 
DBT_PACKAGES_FOLDER
environment variable to a location where your specific user has write permissions:
Copy code
ENV DBT_PACKAGES_FOLDER=/home/your_user/dbt_packages
2. If you're installing Elementary in your Dockerfile, add the permission change after installation:
Copy code
RUN pip install elementary-data && \        chown -R your_user:your_group /usr/local/lib/python3.11/site-packages/elementary
3. Pre-install the dependencies during build:
Copy code
RUN cd $(python -c "import os; from elementary.monitor import dbt_project_utils; print(os.path.dirname(os.path.realpath(dbt_project_utils.__file__)))")/dbt_project && dbt deps
These solutions should work regardless of whether you're using a dbt base image or another base image, as long as dbt and Elementary are properly installed.
d
@kapa.ai for the 3. do I need to do that as a root or after I switch as dbt user ?
k
A message from kapa.ai
For option 3 (pre-installing dependencies during build), you should run the command as the user that has write permissions to the site-packages directory. In most Docker setups, this would be as root during the build phase, before you switch to your dbt user. The typical pattern in a Dockerfile would be:
Copy code
# Install elementary as root RUN pip install elementary-data  # Pre-install the dependencies as root RUN cd $(python -c "import os; from elementary.monitor import dbt_project_utils; print(os.path.dirname(os.path.realpath(dbt_project_utils.__file__)))")/dbt_project && dbt deps  # Then switch to your non-root user USER dbt_user
This approach works because the dependencies are installed during the Docker image build process when you typically have root privileges. Once the dependencies are installed, Elementary won't try to reinstall them when running as the dbt user. This pattern has been successful for users in production environments as seen in the [Slack forum discussions](https://slack.com/archives/C02CTC89LAX/p1686278697336859), where they run the 
dbt deps
command for Elementary's internal project during the Docker build phase.
d
@kapa.ai I have an edr error; 
[2025-04-12, 05:16:33 UTC] {standard_task_runner.py:110} ERROR - Failed to execute job 1760368 for task run-clusterV2 (Pod test-98yc8h2d returned a failure.
failed to create containerd task: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: exec: "edr": executable file not found in $PATH: unknown
remote_pod: {'api_version': 'v1',
k
A message from kapa.ai
I can see you're encountering an error where the 
edr
command is not found in your container's PATH. This is a common issue when trying to run Elementary Data's CLI in containerized environments. The error message 
executable file not found in $PATH: unknown
indicates that your container can't locate the 
edr
command when trying to execute it. This typically happens for one of these reasons: ## Possible causes and solutions: ### 1. Elementary Data is not installed Make sure Elementary Data is properly installed in your container:
Copy code
pip install elementary-data
For your specific data warehouse, you may need to install the appropriate adapter:
Copy code
pip install 'elementary-data[snowflake]'  # or bigquery, redshift, etc.
### 2. Python scripts directory is not in PATH When Elementary is installed via pip, the 
edr
executable is placed in Python's scripts directory, which might not be in your container's PATH. This is a common issue as seen in multiple [Slack forum discussions](https://slack.com/archives/C02CTC89LAX/p1647337659883589). Add the Python scripts directory to your PATH in your Dockerfile:
Copy code
ENV PATH="/usr/local/bin:/usr/local/lib/python3.11/site-packages:/usr/local/lib/python3.11/bin:${PATH}"
### 3. Using the wrong command in Airflow/Kubernetes If you're using Airflow or Kubernetes, make sure you're using the full path to the 
edr
command or setting the PATH correctly in your task definition. As seen in [Slack discussions](https://slack.com/archives/C02CTC89LAX/p1691513076109829), you might need to specify:
Copy code
PATH=$PATH:/usr/local/bin edr ...
### 4. Verify installation You can verify if Elementary is properly installed by running:
Copy code
which edr
If this doesn't return a path, Elementary is not properly installed or not in your PATH. For containerized environments, I recommend explicitly setting the PATH in your Dockerfile and verifying the installation during the build process to ensure everything is set up correctly before deploying to production. Let me know if you need more specific guidance based on your container setup!
28 Views
Open in Slack
PreviousNext
Powered by