You’re hitting a permissions issue because

edr report

always runs

dbt deps

for Elementary’s internal dbt project (under

site-packages/elementary/monitor/dbt_project

). On Databricks (especially shared clusters / dbt-tasks), that location is often read‑only, so

dbt deps

can’t create

dbt_packages

or

package-lock.yml

and fails with

PermissionError: [Errno 13] Permission denied

. [[Databricks shared issue](https://github.com/elementary-data/elementary/issues/1584); [Slack Databricks perms](https://elementary-community.slack.com/archives/C02CTC89LAX/p1718142965337009?thread_ts=1718142965.337009&cid=C02CTC89LAX)] From the shared solutions, you have two practical ways to avoid this in Databricks/dbt-tasks: --- ### 1. Redirect where Elementary installs internal dbt packages Elementary’s CLI honors a

DBT_PACKAGES_FOLDER

env var for its internal project; you can point it to a writable path (e.g. under

/tmp

). [[DBT_PACKAGES_FOLDER hint](https://elementary-community.slack.com/archives/C02CTC89LAX/p1716798424976069?thread_ts=1716798424.976069&cid=C02CTC89LAX)] In your Databricks job task (or notebook cell before calling

edr

):

bash export DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages edr report -p . -t elementary

On Databricks notebooks, that would typically be:

bash %sh export DBT_PACKAGES_FOLDER=/tmp/edr_dbt_packages edr report -p . -t elementary

Make sure

/tmp

(or whatever you choose) is writable for the job user. This avoids writing into

/local_disk0/.ephemeral_nfs/.../site-packages/elementary/monitor/dbt_project/dbt_packages

or

package-lock.yml

, which is what causes the permission error. [[Databricks package-lock error](https://github.com/elementary-data/elementary/issues/1584)] --- ### 2. Ensure the Elementary install directory is writable (less ideal on Databricks) In containerized environments, some teams

chown

the Elementary install path so the non‑root user can write there, allowing

dbt deps

to succeed. [[Docker chown fix](https://elementary-community.slack.com/archives/C02CTC89LAX/p1674851653694939?thread_ts=1674851653.694939&cid=C02CTC89LAX)] On Databricks managed clusters this is usually not feasible or recommended, so the env‑var approach above is the safer pattern. --- If you still see

Permission denied: 'dbt_packages'

after setting

DBT_PACKAGES_FOLDER

, double‑check: - The env var is set in the same process / task that runs

edr report

. - The folder exists and is writable by the dbt-task user.

That new error is coming from dbt itself (Elementary just wraps it as

DbtCommandError

), and the specific

[Errno 39] Directory not empty: 'configuration'

message is not documented in the Elementary Databricks threads and issues you shared. The existing Databricks errors we do see are: -

Permission denied: 'dbt_packages'

/

Invalid argument: 'dbt_packages'

when dbt tries to create its

dbt_packages

directory in a non‑writable or invalid location. [[Databricks GC/Invalid argument](https://elementary-community.slack.com/archives/C02CTC89LAX/p1691413157144939?thread_ts=1691413157.144939&cid=C02CTC89LAX); [GCS Airflow perms](https://elementary-community.slack.com/archives/C02CTC89LAX/p1681911667304009?thread_ts=1681911667.304009&cid=C02CTC89LAX)] - Permission errors writing in

site-packages/elementary/monitor/dbt_project

on shared clusters. [[Generate report fails – Databricks shared](https://github.com/elementary-data/elementary/issues/1584)] None of the knowledge sources mention a

[Errno 39] Directory not empty: 'configuration'

failure, so I can’t say with confidence exactly what directory dbt is trying to modify or why. Given that: 1. This is still a dbt command failing underneath

edr report

. [[DbtCommandError examples](https://github.com/elementary-data/elementary/issues/1931)] 2. The error type (directory not empty) suggests some filesystem constraint in your Databricks job environment, similar in flavor to the earlier permission/invalid‑path issues, but on a different path. I’d recommend the next steps that are supported in the docs and threads: 1. Inspect

edr.log

and

dbt.log

In every similar case, support asks for the logs from the

edr_target

directory to see the full dbt command and exact path that’s failing. [[Ask for edr.log](https://elementary-community.slack.com/archives/C02CTC89LAX/p1695913966874839?thread_ts=1695913966.874839&cid=C02CTC89LAX); [location of edr.log](https://elementary-community.slack.com/archives/C02CTC89LAX/p1697659818900659?thread_ts=1697659818.900659&cid=C02CTC89LAX)] 2. Confirm you’re pointing Elementary to the right, writable dirs From the existing Databricks discussions, two patterns matter: - Ensure the paths you pass to

--profiles-dir

(and, if you use it,

--project-dir

) are concrete directories, not their parent. E.g. use

--profiles-dir .dbt

instead of the parent folder. [[Profiles dir fix](https://elementary-community.slack.com/archives/C02CTC89LAX/p1742287892945519?thread_ts=1742287892.945519&cid=C02CTC89LAX)] - Use a writable temp/work directory (e.g. under

/tmp

) for any Elementary/ dbt artifacts (packages, targets). The Databricks shared‑cluster thread and the recent “Config tries to create

edr_target

in a non‑writable repo” case both show that when the default location is not writable, you must change working dir or configuration so Elementary writes to a writable path. [[Generate report fails – Databricks shared](https://github.com/elementary-data/elementary/issues/1584); [edr_target in read‑only repo](https://elementary-community.slack.com/archives/C02CTC89LAX/p1740037553539379?thread_ts=1740037553.539379&cid=C02CTC89LAX)] 3. If you can, reproduce outside the dbt‑task pipeline Several users narrowed down environment issues by: - Running

edr report

in a local or notebook context with a clean virtualenv and correct

profiles.yml

, which then worked; their failures turned out to be environment/permissions issues in the orchestrated context. [[Clean reinstall fixed](https://elementary-community.slack.com/archives/C02CTC89LAX/p1692717229853259?thread_ts=1692717229.853259&cid=C02CTC89LAX); [venv + fresh install advice](https://elementary-community.slack.com/archives/C02CTC89LAX/p1697659818900659?thread_ts=1697659818.900659&cid=C02CTC89LAX)] Because

[Errno 39] Directory not empty: 'configuration'

itself is not in the knowledge sources, I can’t offer a specific, documented workaround beyond these general steps. The most actionable next move is to: - Grab the failing dbt command and full traceback from

edr_target/edr.log

in that Databricks job run, and - Share it in the Elementary Slack or GitHub issue, as done in the other Databricks error threads. [[Ask community with logs](https://github.com/elementary-data/elementary/issues/1584)]