Using dbt on databricks. When and why does element...
# ask-elementary-ai
f
Using dbt on databricks. When and why does elementary access the defult databricks dbfs storage?
k
A message from kapa.ai
Elementary may access the default Databricks DBFS storage in a few scenarios, primarily related to how dbt and the Databricks adapter handle schema and table creation, as well as how Elementary manages its internal files and dependencies. ### 1. Temporary Table Creation and Default Schema When running dbt (and thus Elementary) on Databricks, there have been reports that temporary tables or even some artifact tables are created in the
default
schema if a custom schema is not properly configured or if the dbt-databricks adapter defaults to it. This is due to the adapter's behavior, not specifically Elementary, but it affects how and where Elementary's tables are created. A fix was introduced in Elementary to address this, but some users have still observed temp tables being created in the
default
schema, especially if the schema configuration is not set correctly or if there are edge cases in the adapter's implementation. This can result in Elementary writing to the default DBFS location associated with the
default
schema in Databricks[is there a way to force elemantary to use an existing schema instead of default schema?](https://slack.com/archives/C02CTC89LAX/p1683819585181719), [Community Member 1106 observation](https://slack.com/archives/C02CTC89LAX/p1697207730746439). ### 2. Internal File and Directory Management Elementary CLI uses an internal dbt project located within its Python package directory (e.g.,
site-packages/elementary/monitor/dbt_project
). When you run EDR commands, Elementary attempts to install dbt dependencies and create directories such as
dbt_packages
and
edr_target
. If the working directory is on a read-only filesystem (which can happen in some Databricks environments), or if the default location is not writable, you may encounter errors. Users have resolved these issues by changing the working directory to a writable location such as
/tmp/elementary
instead of the default DBFS or other restricted paths. This is not about accessing your data, but about where Elementary tries to write its own operational files and dependencies[Community Member 67 explanation](https://slack.com/archives/C02CTC89LAX/p1718142965337009), [Community Member 583 solution](https://slack.com/archives/C02CTC89LAX/p1707952069729759). ### 3. Schema and Storage Location Configuration If the schema for Elementary artifacts is not explicitly set in your dbt project configuration, dbt (and thus Elementary) may default to using the
default
schema, which is mapped to the default DBFS storage location. Ensuring that the schema is set correctly in your
dbt_project.yml
and not overridden in a way that causes conflicts is important to avoid this behavior[Community Member 1653 discussion](https://slack.com/archives/C02CTC89LAX/p1668792958784389). --- Summary: Elementary accesses the default Databricks DBFS storage mainly when the schema is not properly configured, when the dbt-databricks adapter defaults to the
default
schema, or when its internal file operations are directed to a non-writable or default location. Proper configuration of the schema and working directory can prevent unwanted access to the default DBFS storage. If you encounter issues, ensure your schema is set explicitly and use a writable directory like
/tmp/elementary
for operational files.