https://linen.dev logo
Join Slack
Powered by
Why even when I declare the schema to be used as a...
# ask-elementary-ai
l
Why even when I declare the schema to be used as a suffix the elementary still search for the default schema when building views? e.g: 191558 1 of 34 ERROR creating sql view model dummy_dbt_elementary.alerts_anomaly_detection [ERROR in 0.39s] 191558 2 of 34 START sql view model dummy_dbt_elementary.alerts_dbt_source_freshness .. [RUN] 191558 2 of 34 ERROR creating sql view model dummy_dbt_elementary.alerts_dbt_source_freshness [ERROR in 0.40s] 191558 3 of 34 START sql view model dummy_dbt_elementary.alerts_dbt_tests ............. [RUN] 191559 3 of 34 ERROR creating sql view model dummy_dbt_elementary.alerts_dbt_tests .... [ERROR in 0.58s] 191559 4 of 34 START sql view model dummy_dbt_elementary.alerts_schema_changes ........ [RUN] 191559 4 of 34 ERROR creating sql view model dummy_dbt_elementary.alerts_schema_changes [ERROR in 0.38s] 191559 5 of 34 START sql incremental model dummy_dbt_elementary.data_monitoring_metrics [RUN] 191616 5 of 34 OK created sql incremental model dummy_dbt_elementary.data_monitoring_metrics [OK in 16.54s] 191616 6 of 34 START sql view model dummy_dbt_elementary.dbt_artifacts_hashes ......... [RUN] 191617 6 of 34 ERROR creating sql view model dummy_dbt_elementary.dbt_artifacts_hashes [ERROR in 0.83s] 191617 7 of 34 START sql incremental model dummy_dbt_elementary.dbt_columns ........... [RUN] 191628 7 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_columns ...... [OK in 11.67s] 191628 8 of 34 START sql incremental model dummy_dbt_elementary.dbt_exposures ......... [RUN] 191639 8 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_exposures .... [OK in 10.77s] 191639 9 of 34 START sql incremental model dummy_dbt_elementary.dbt_groups ............ [RUN] 191647 9 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_groups ....... [OK in 8.13s] 191647 10 of 34 START sql incremental model dummy_dbt_elementary.dbt_invocations ...... [RUN] 191656 10 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_invocations . [OK in 8.77s] 191656 11 of 34 START sql incremental model dummy_dbt_elementary.dbt_metrics .......... [RUN] 191708 11 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_metrics ..... [OK in 11.67s] 191708 12 of 34 START sql incremental model dummy_dbt_elementary.dbt_models ........... [RUN] 191720 12 of 34 OK created sql incremental model dummy_dbt_elementary.dbt_models ...... [OK in 11.70s] 191720 13 of 34 START sql incremental model dummy_dbt_elementary.dbt_run_results ...... [RUN] 192034 Failure in model alerts_anomaly_detection (models/edr/alerts/alerts_anomaly_detection.sql) 192034 Database Error in model alerts_anomaly_detection (models/edr/alerts/alerts_anomaly_detection.sql) [TABLE_OR_VIEW_NOT_FOUND] The table or view 
platform
.
dummy_dbt
.
elementary_test_results
cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog. To tolerate the error on drop use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS. SQLSTATE: 42P01; line 4 pos 18 192034 192034 compiled code at target/compiled/elementary/models/edr/alerts/alerts_anomaly_detection.sql
k
A message from kapa.ai
Based on the error you're experiencing and the knowledge sources, this is a common issue related to custom schema configurations in Elementary. The problem occurs when you have a custom 
generate_schema_name
macro in your dbt project that affects how Elementary creates its tables. ## The Issue Elementary is trying to find tables in 
platform.dummy_dbt.elementary_test_results
(your default schema), but the tables were actually created in 
dummy_dbt_elementary
(with the custom schema suffix). This mismatch happens because: 1. dbt commands use your custom 
generate_schema_name
macro to create Elementary tables 2. edr commands look for tables based on the schema specified in your 
profiles.yml
under the 
elementary
profile, which doesn't respect your custom schema logic ## The Solution You need to exclude Elementary from your custom 
generate_schema_name
macro. Here's how to fix it, as explained in [this Slack discussion](https://slack.com/archives/C02CTC89LAX/p1712851926036669):
Copy code
jinja {% macro generate_schema_name(custom_schema_name, node) -%}     {% if node.package_name == 'elementary' %}         {% set schema_name = node.package_name | trim %}     {% else %}         {# your existing custom logic here #}     {% endif %}     {{ schema_name }} {%- endmacro %}
Or, if you want Elementary to follow dbt's default behavior while keeping your custom logic for other models:
Copy code
jinja {% macro generate_schema_name(custom_schema_name, node) -%}     {% if node.package_name == 'elementary' and custom_schema_name %}        {{ target.schema }}_{{ custom_schema_name | trim }}     {% else %}        {# your existing custom logic #}     {% endif %} {%- endmacro %}
## Additional Configuration Make sure your 
profiles.yml
has an 
elementary
profile pointing to the correct schema where Elementary tables are actually created. Based on your error, it should be:
Copy code
yaml elementary:   outputs:     default:       # ... other configs       schema: dummy_dbt_elementary  # or whatever schema Elementary tables are in
After making these changes, run 
dbt run --select elementary
again to recreate the Elementary models in the correct schema.
l
@kapa.ai My project's configuration is like this: dbt_project.yml: elementary: +tags: ['service_name=elementary-test','owner_layer_slug=data-platform','data_domain_layer_slug=plat'] +schema: elementary enabled: "{{ target.name in ('prod') }}" profiles.yml: prod: type: databricks method: http catalog: platform schema: dummy_dbt my custom macro: {% macro default__generate_schema_name(custom_schema_name, node) -%} {%- set default_schema = target.schema -%} {%- if custom_schema_name is none or custom_schema_name | trim == '' -%} {{ default_schema }} {%- elif custom_schema_name | trim == 'general_elementary_metrics'-%} {{ custom_schema_name | trim }} {%- else -%} {{ default_schema }}_{{ custom_schema_name | trim }} {%- endif -%} {%- endmacro %}
k
A message from kapa.ai
Based on your configuration, I can see the issue. Your custom 
generate_schema_name
macro is correctly configured to create Elementary tables in 
dummy_dbt_elementary
(since 
custom_schema_name
is "elementary" and your macro appends it to the default schema). However, you need to ensure your 
profiles.yml
reflects where the Elementary tables are actually created. ## The Problem Your 
profiles.yml
currently has:
Copy code
yaml schema: dummy_dbt
But Elementary tables are being created in 
dummy_dbt_elementary
due to your custom macro logic. When 
edr
commands run, they only use the 
profiles.yml
configuration and don't respect your custom 
generate_schema_name
macro, as explained in [this GitHub issue](https://github.com/elementary-data/elementary/issues/1462). ## The Solution You need to add a separate 
elementary
profile in your 
profiles.yml
that points to the actual schema where Elementary tables exist:
Copy code
yaml elementary:   outputs:     prod:       type: databricks       method: http       catalog: platform       schema: dummy_dbt_elementary  # This should match where your tables are actually created       # ... other connection details   target: prod
As shown in [this Slack discussion](https://slack.com/archives/C02CTC89LAX/p1726599930360009), the 
elementary
profile should be a top-level profile (same indentation as your existing dbt profile). ## Alternative Approach If you want to exclude Elementary from your custom schema logic entirely, you could modify your macro to handle Elementary specifically:
Copy code
jinja {% macro default__generate_schema_name(custom_schema_name, node) -%}     {%- set default_schema = target.schema -%}     {%- if node.package_name == 'elementary' and custom_schema_name -%}         {{ default_schema }}_{{ custom_schema_name | trim }}     {%- elif custom_schema_name is none or custom_schema_name | trim == '' -%}         {{ default_schema }}     {%- elif custom_schema_name | trim == 'general_elementary_metrics'-%}         {{ custom_schema_name | trim }}     {%- else -%}         {{ default_schema }}_{{ custom_schema_name | trim }}     {%- endif -%} {%- endmacro %}
After making these changes, run 
dbt run --select elementary
to ensure the tables are in the correct location, and then your 
edr
commands should work properly.
l
@kapa.ai I'm in dbt_models but is saving data from the elementary itself, does It really need to do this? model.elementary_test.orders_daily orders_daily f4e274cd0ead8fc8ff9314e05371ce2465dfa0668182a79ba3c786f802d55e32 view [] {"data_domain_layer_slug": "plat", "owner_layer_slug": "data-platform", "owners": "luiz_felipe_campos_da_silva", "service_name": "elementary-test", "quality": "silver"} [] platform dummy_dbt ["macro.macros_nexus.apply_table_meta"] ["model.elementary_test.orders_raw"] Number of orders by day orders_daily elementary_test src/models/example/orders_daily.sql example/orders_daily.sql elementary_test://src/models/example/schema.yml 2025-11-03 175739 c5fed350aea87aa22bc96edb285479d4 null null null protected model.elementary_test.orders_raw orders_raw 9d2a3b4f9008a53eb597e2d682b8c0a56d4b9e0a0a406bfc53f7a3ffea13cd44 view [] {"data_domain_layer_slug": "plat", "owner_layer_slug": "data-platform", "owners": "luiz_felipe_campos_da_silva", "service_name": "elementary-test", "quality": "silver"} [] platform dummy_dbt ["macro.macros_nexus.apply_table_meta"] [] Raw ingested orders orders_raw elementary_test src/models/example/orders_raw.sql example/orders_raw.sql elementary_test://src/models/example/schema.yml 2025-11-03 175739 3b7ec9bedb9cb6bc9ed5b36cfd2f3893 null null null protected model.elementary.elementary_test_results elementary_test_results df74ce864e711c45b8f20af4d3246257f95ef0735e17d83dce220bdd80bde93a incremental ["data_domain_layer_slug=plat", "owner_layer_slug=data-platform", "service_name=elementary-test"] {"timestamp_column": "created_at", "prev_timestamp_column": "detected_at"} [] platform dummy_dbt_elementary ["macro.elementary.empty_elementary_test_results", "macro.elementary.get_config_var", "macro.elementary.get_default_incremental_strategy", "macro.elementary.get_default_table_type"] [] "Run results of all dbt tests, with fields and metadata needed to produce the Elementary report UI. Each row is the result of a single test, including native dbt tests, packages tests and elementary tests. New data is loaded to this model on an on-run-end hook named 
elementary.handle_tests_results
. " elementary_test_results elementary models/edr/run_results/elementary_test_results.sql edr/run_results/elementary_test_results.sql elementary://models/run_results.yml 2025-11-03 175739 066f85350eb21c40078f633b674efbba id null null protected model.elementary.dbt_source_freshness_results dbt_source_freshness_results fb063c8eb524356395464854c25d33038742ce2e0c3d4b3936ce8919a0ee27d6 incremental ["data_domain_layer_slug=plat", "owner_layer_slug=data-platform", "service_name=elementary-test"] {"timestamp_column": "created_at", "prev_timestamp_column": "generated_at"} [] platform dummy_dbt_elementary ["macro.elementary.empty_dbt_source_freshness_results", "macro.elementary.get_config_var", "macro.elementary.get_default_incremental_strategy", "macro.elementary.get_default_table_type"] [] dbt_source_freshness_results elementary models/edr/run_results/dbt_source_freshness_results.sql edr/run_results/dbt_source_freshness_results.sql null 2025-11-03 175739 a2279649167b401c130e34c05121b932 source_freshness_execution_id null null protected
k
A message from kapa.ai
Based on the data you're showing from the 
dbt_models
table, yes, Elementary does need to save information about its own models. This is expected behavior. ## Why Elementary Saves Its Own Model Information The 
dbt_models
table is one of Elementary's metadata models that stores information about all models in your dbt project, including Elementary's own models. As explained in [this Slack discussion](https://slack.com/archives/C02CTC89LAX/p1673857363237629): > Metadata models (
dbt_sources
, 
dbt_models
, etc) are regular tables, so we use 
truncate and insert
to update to the current project state. This means that every time you run 
dbt run --select elementary
, Elementary updates the 
dbt_models
table with the current state of all models in your dbt project's manifest, which includes: - Your project's models (
orders_daily
, 
orders_raw
) - Elementary's own models (
elementary_test_results
, 
dbt_source_freshness_results
) ## Why This Is Necessary Elementary needs to track its own models because: 1. Complete lineage tracking: Elementary provides lineage and dependency tracking across your entire dbt project, including how its own models relate to your data 2. Metadata consistency: The 
dbt_models
table reflects the complete state of your dbt project as captured in the manifest.json 3. Internal operations: Elementary uses this metadata for its monitoring and reporting features This is normal and expected behavior. The Elementary models you see in the 
dbt_models
table are the ones created in your 
dummy_dbt_elementary
schema, which is correct based on your configuration.
35 Views
Open in Slack
PreviousNext
Powered by