<@U088WCP3B4P> We have a few clients now each with...
# ask-elementary-ai
r
@kapa.ai We have a few clients now each with their own sets of dbt projects that have at least 1 schema for up to 3 environments. Adding another for elementary is ballooning the list of schemas even more. Is there a common practice to either cut down on the number of schemas or putting them somewhere else? We are using databricks jobs to run dbt and the data is going into databricks sql. I know we can set it up to only run for production but obviously it's nice to see the process work in lower environments first.
k
A message from kapa.ai
You’re running into a limitation that comes up a lot with multi‑project / multi‑env setups: Elementary OSS today is designed around “one Elementary schema per dbt project (per target/env)”, and there isn’t an officially supported way to safely collapse multiple dbt projects into a single Elementary schema without conflicts. From the sources: - Elementary explicitly recommends a separate Elementary schema per dbt project to avoid artifacts overwriting each other. [[Multi‑project thread](https://elementary-community.slack.com/archives/C02CTC89LAX/p1693465315902869); [Another multi‑project thread](https://elementary-community.slack.com/archives/C02CTC89LAX/p1698782900980109)] - Each Elementary schema effectively represents a single dbt project, and metadata tables are reloaded on each run, so sharing a schema across projects causes overrides. [[Per‑project schema](https://elementary-community.slack.com/archives/C02CTC89LAX/p1741876922074969)] - For multiple environments in a single dbt project, you can keep one Elementary schema per project and just use multiple targets; Elementary handles multiple jobs/schemas within that project fine. [[Multi‑job same project](https://elementary-community.slack.com/archives/C02CTC89LAX/p1699906892923319); [Single schema per project](https://elementary-community.slack.com/archives/C02CTC89LAX/p1686252653101959)] Given that, the “common practice” in OSS, based on the docs and Slack answers, is: 1. Accept one Elementary schema per dbt project (and per env/target if you want separate monitoring per env). 2. If you want to limit schema sprawl, only enable Elementary in selected environments (often prod, sometimes staging) using conditional config, e.g.
+enabled: "{{ target.name == 'prod' }}"
. [[Disable per env](https://elementary-community.slack.com/archives/C02CTC89LAX/p1675438839754619); [FAQ disablement](https://docs.elementary-data.com/oss/general/faq#page-title)] 3. Optionally, disable hooks / uploads in lower envs via vars so you can still run dbt there without creating more Elementary schemas or data: [[Disable hooks](https://docs.elementary-data.com/oss/general/faq#page-title)]
Copy code
yaml    vars:      disable_run_results: "{{ target.name not in ['prod','staging'] }}"      disable_tests_results: "{{ target.name != 'prod' }}"      disable_dbt_artifacts_autoupload: "{{ target.name != 'prod' }}"      disable_dbt_invocation_autoupload: "{{ target.name != 'prod' }}"
There are a few workarounds people have experimented with, but they’re not officially supported and come with caveats: - Unioning tables from multiple Elementary schemas into a “union_all_elementary” schema and pointing the CLI at that, mainly for consolidated reporting/metrics. This has been suggested as a stopgap, but requires you to manage the unions yourself and doesn’t change the need for per‑project schemas underneath. [[Union stopgap](https://elementary-community.slack.com/archives/C02CTC89LAX/p1673857363237629); [Union for cross‑customer stats](https://elementary-community.slack.com/archives/C02CTC89LAX/p1736169272612519)] - Some users tried to share a single schema across projects and ran into overrides; Elementary explicitly says this is not supported today. [[Not supported multi‑project one schema](https://elementary-community.slack.com/archives/C02CTC89LAX/p1693465315902869); [Override explanation](https://elementary-community.slack.com/archives/C02CTC89LAX/p1673857363237629)] For Cloud, there is a way to consolidate views across multiple dbt projects and environments in one UI while still keeping separate Elementary schemas per project/env; that’s handled at the environment level in the app, not by reducing schemas in your warehouse. [[Cloud multi‑env](https://docs.elementary-data.com/cloud/features/multi-env)] So, based on the available info: - There isn’t a documented, supported pattern to significantly cut down the number of Elementary schemas across different dbt projects. - The practical levers you have are: - Only enable Elementary (or its hooks) in the environments where you truly need observability. - Keep one Elementary schema per dbt project and avoid sharing it across projects. - If you need a unified view, consider Cloud’s multi‑environment support rather than collapsing schemas in Databricks. [[Cloud multi‑env](https://docs.elementary-data.com/cloud/features/multi-env)] If you share how many projects/envs you have per client (e.g., 3 projects × 3 envs), I can suggest a concrete pattern for which envs to enable and how to name schemas to keep things manageable.