@kapa.ai I'm running edr report from a Databricks job (Asset Bundle deployment) and hitting a chain of issues with Elementary 0.23.4. I want to confirm I'm on the supported path and ask about a few specifics that aren't clear from the docs or GitHub issues. Setup - Databricks (Azure), Asset Bundle deploys job to /Workspace/Shared/.bundle/... - Task: spark_python_task running a Python script that calls edr report --profiles-dir <vol>/profile --target-path <vol> --project-dir <project> --open-browser false --env <env> - elementary-data[databricks] is installed as a cluster library (via our wheel's transitive deps), unpinned, so pip resolves to 0.23.4 - dbt-core 1.10.11, dbt-databricks 1.10.12 - Cluster was previously data_security_mode: DATA_SECURITY_MODE_STANDARD (shared access) What broke 1. First error — the user's --project-dir had no dbt_packages/ folder. Elementary 0.23 reads <project_dir>/dbt_packages/elementary/dbt_project.yml in is_dbt_package_up_to_date and raises FileNotFoundError. I fixed this by materializing a writable copy of the dbt project on a Volume and running dbt deps against it before invoking edr. 2. Second error — Elementary's internal DbtRunner (the one constructed inside DataMonitoringReport.init via _init_internal_dbt_runner) ran dbt deps against its own bundled project at .../site-packages/elementary/monitor/dbt_project/ and got PermissionError: [Errno 13] Permission denied: 'dbt_packages'. I've found GitHub issue #1584 documenting this on shared clusters and I'm moving the job to a single-user cluster. Questions 1. Is single-user cluster the long-term supported deployment path for edr report on Databricks, or is there an officially recommended pattern (e.g. notebook-level %pip install, a published Docker image, a Databricks app) you'd point production users at instead? 2. Is there any way to disable or skip the internal dbt deps step at runtime via env var, CLI flag, or config — for environments where the internal project's dbt_packages can be pre-baked? I'd prefer this over relying on writable site-packages. 3. Can the internal dbt project location be redirected to a writable path via env var or config? I see Path(elementary.file).parent / "monitor" / "dbt_project" is hardcoded; is there a supported override? 4. Is there a recommended version pin for elementary-data when used as a Databricks cluster library in 2026? We had been on 0.20.1 implicitly and it "just worked"; the unpinned upgrade to 0.23.x exposed new behaviors. Should we be tracking a specific release line, and is there a published compatibility matrix vs dbt-core / dbt-databricks? 5. For the user's dbt project, what does Elementary actually need installed in <project_dir>/dbt_packages/? Just elementary itself (the dbt package, not the pip), or does it also need dbt_utils and other transitive deps materialized? I want to make my pre-dbt deps step minimal and reliable. 6. Are there any known Elementary 0.23.x behaviors I should be aware of that didn't exist in 0.20.x — schema migrations, additional Volume writes, new permission requirements on the underlying tables? If you have a canonical "Elementary on Databricks production deployment" guide that addresses these, a link is fine.

These are detailed and specific questions. I can address some of them from the knowledge sources, but I must be upfront: the knowledge sources do not cover Elementary 0.23.x specifically, version compatibility matrices, or a canonical "Elementary on Databricks production deployment" guide. I'll answer what I can and clearly flag gaps. Question 1: Supported deployment path on Databricks The knowledge sources do not document an "officially supported" deployment path. What the community has established: • Single-user cluster does work (the

site-packages

path is writable for the job user). [Databricks shared issue] • Shared/standard cluster consistently fails with permission errors on the internal dbt project path — this is a known, documented issue with no official fix. [Databricks shared issue] • Notebook-level

%pip install

also works because it installs into a user-writable location, but is noted as impractical for wheel-based deployments. [Databricks shared issue] • No published Docker image or Databricks App pattern is mentioned in the knowledge sources. Moving to a single-user cluster is the most commonly cited working path, but it is not described as an "official recommendation" in any source I have. Question 2: Skipping or pre-baking the internal dbt deps step There is no documented CLI flag or env var to skip the internal

dbt deps

step entirely. The closest options are: •

DBT_PACKAGES_FOLDER

env var: redirects where

dbt deps

installs packages for the internal project to a writable path (e.g.

/tmp/edr_dbt_packages

). [DBT_PACKAGES_FOLDER hint] •

--update-dbt-package false

CLI flag: mentioned in one thread as potentially helping, though not fully confirmed. [permission workarounds] • Pre-running

dbt deps

against the internal project path before invoking

edr

(e.g. in a cluster init script or setup cell):

dbt --log-format json deps \
  --project-dir /local_disk0/.ephemeral_nfs/.../site-packages/elementary/monitor/dbt_project

[Pre-deps pattern] None of these fully eliminate the runtime

dbt deps

call in all cases. Question 3: Redirecting the internal dbt project to a writable path The knowledge sources confirm that: •

DBT_PACKAGES_FOLDER

redirects the

dbt_packages

install directory. [DBT_PACKAGES_FOLDER hint] •

DBT_TARGET_PATH

has been used by community members to redirect the

target

directory, though whether it is fully honoured is not confirmed in the knowledge sources. [env var approach] •

package-lock.yml

is written directly inside the internal project directory (not inside

dbt_packages

), so

DBT_PACKAGES_FOLDER

alone does not prevent all permission errors. [package-lock issue] • There is no supported env var or config to relocate the entire internal dbt project path. The path being hardcoded is a known limitation, and there is no documented override. [known limitation] The most reliable workaround for read-only

site-packages

is `chown`/`chmod` on the Elementary install directory, or installing into a writable venv. [Docker chown fix] Question 4: Version pinning and compatibility matrix The knowledge sources do not contain a compatibility matrix for

elementary-data

vs

dbt-core

/

dbt-databricks

, and do not document recommended version pins for 2026 or the 0.23.x line. I cannot answer this reliably from the available sources. I'd recommend checking the Elementary GitHub releases and the official docs directly. Question 5: What does Elementary need in project_dir/dbt_packages/? The knowledge sources do not explicitly describe what transitive dbt package dependencies Elementary requires materialized in the user project's

dbt_packages/

. They only describe the internal project's own

dbt deps

step. I cannot give you a reliable minimal list from these sources — running

dbt deps

against your project's

packages.yml

(which references the

elementary

dbt package) is the documented approach, and that will resolve transitive deps like

dbt_utils

automatically. Question 6: Known 0.23.x behavioral changes vs 0.20.x The knowledge sources do not cover Elementary 0.23.x specifically or document behavioral differences from 0.20.x (schema migrations, new Volume writes, new permission requirements). I cannot answer this from the available sources. Summary of what's confirmed vs uncertain For questions 4–6 and an official production guide, I'd recommend opening a discussion on the Elementary GitHub or reaching out in the Elementary Slack community directly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements