https://linen.dev logo
Join Slack
Powered by
I'm developing a dbt package. Our dbt package need...
# ask-elementary-ai
c
I'm developing a dbt package. Our dbt package needs to support a database called fabric. I'm using a package called elementary in dbt. I tried to disabled it but that does not work. This package needs to be used in every other databse but fabric. When I run the command ⁠dbt build --full-refresh, I get the following output: (.venv) demo % dbt build --full-refresh -x 172623 Running with dbt=1.8.6 172623 Registered adapter: fabric=1.9.3 172624 Found 862 models, 97 seeds, 3 operations, 134 data tests, 1610 macros 172624 172627 172627 Running 1 on-run-start hook 172627 IMPORTANT - Starting from dbt 1.8, users must explicitly allow packages to override materializations. Elementary requires this ability to support collection of samples and failed row count for dbt tests. Please add the following flag to dbt_project.yml to allow it: flags: require_explicit_package_overrides_for_builtin_materializations: false Notes - * This is a temporary measure that will result in a deprecation warning, please ignore it for now. Elementary is working with the dbt-core team on a more permanent solution. * This message can be muted by setting the 'mute_ensure_materialization_override' var to true. 172627 1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN] 172627 1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s] 172627 172627 Concurrency: 4 threads (target='fabric') 172627 172627 1 of 1045 START sql incremental model dbt_cjones.data_monitoring_metrics ....... [RUN] 172627 2 of 1045 START sql incremental model dbt_cjones.dbt_columns ................... [RUN] 172627 3 of 1045 START sql incremental model dbt_cjones.dbt_exposures ................. [RUN] 172627 4 of 1045 START sql incremental model dbt_cjones.dbt_invocations ............... [RUN] 172628 4 of 1045 ERROR creating sql incremental model dbt_cjones.dbt_invocations ...... [ERROR in 0.37s] 172628 5 of 1045 START sql incremental model dbt_cjones.dbt_metrics ................... [RUN] 172628 CANCEL query model.elementary.data_monitoring_metrics .......................... [CANCEL] 172628 CANCEL query model.elementary.dbt_columns ...................................... [CANCEL] 172628 CANCEL query model.elementary.dbt_exposures .................................... [CANCEL] 172628 CANCEL query model.elementary.dbt_invocations .................................. [CANCEL] Completed with 1094 errors and 0 warnings: 172640 172640 Database Error in model dbt_invocations (models/edr/dbt_artifacts/dbt_invocations.sql) ('42S22', "[42S22] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid column name 'True'. (207) (SQLExecDirectW)") compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_invocations.sql 172639 Database Error in model dbt_invocations (models/edr/dbt_artifacts/dbt_invocations.sql) ('42S22', "[42S22] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid column name 'True'. (207) (SQLExecDirectW)") compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_invocations.sql Here is the dbt_project.yml WITHIN THE PACKAGE I'm importing into a demo project: name: 'the_tuva_project' version: '0.14.2' config-version: 2 require-dbt-version: ">=1.5.0" profile: default models: elementary: +schema: | {%- if var('tuva_schema_prefix',None) != None -%}{{var('tuva_schema_prefix')}}_elementary{% else %}elementary{%- endif -%} +enabled: "{{ ((target.type != 'fabric') and var('enable_input_layer_testing', true)) | as_bool }}" flags: require_explicit_package_overrides_for_builtin_materializations: false vars: record_type: "ip" dxccsr_version: "'2023.1'" prccsr_version: "'2023.1'" cms_hcc_payment_year: '{{ run_started_at.strftime("%Y") }}' tuva_last_run: '{{ run_started_at.astimezone(modules.pytz.timezone("UTC")) }}' custom_bucket_name: tuva-public-resources ## DQI data testing on input layer enable_input_layer_testing: true on-run-end: - "{{ log_warning_for_seeds() }}" model-paths: ["models"] analysis-paths: ["analyses"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"] docs-paths: ["dbt_doc_blocks"] target-path: "target" clean-targets: - "target" - "dbt_packages" Here is the dbt_project.yml in the demo project itself: name: 'the_tuva_project_demo' version: '0.1.9' config-version: 2 profile: default vars: tuva_schema_prefix: "tuva_dbt_cjones" claims_enabled: true clinical_enabled: true provider_attribution_enabled: true cms_hcc_payment_year: 2018 quality_measures_period_end: "2018-12-31" model-paths: ["models"] analysis-paths: ["analyses"] test-paths: ["tests"] seed-paths: ["seeds"] macro-paths: ["macros"] snapshot-paths: ["snapshots"] target-path: "target" clean-targets: - "target" - "dbt_packages" Can you help me troubleshoot this issue?
k
A message from kapa.ai
I'll help you troubleshoot the issue with Elementary Data in your dbt package when using the Fabric database. Based on the error logs and your configuration, I can see that you're trying to disable Elementary for the Fabric database, but it's still attempting to run and causing errors. The issue appears to be in how you've configured the Elementary package in your dbt_project.yml. Let's look at what's happening: 1. You've set the Elementary models to be disabled when the target type is 'fabric':
Copy code
yaml elementary:   +schema: |    {%- if var('tuva_schema_prefix',None) != None -%}{{var('tuva_schema_prefix')}}_elementary{% else %}elementary{%- endif -%}   +enabled: "{{ ((target.type != 'fabric') and var('enable_input_layer_testing', true)) | as_bool }}"
However, there might be a few issues with this approach: 1. The error suggests that Elementary is still trying to run despite your configuration. 2. The specific error 
Invalid column name 'True'
indicates a SQL syntax issue that might be related to how the boolean value is being handled in the Fabric adapter. Here are some solutions to try: ## Solution 1: Update your Elementary configuration Make sure your Elementary configuration is properly placed in the dbt_project.yml. According to the Elementary documentation, the configuration should be under the 
models
key:
Copy code
yaml models:   elementary:     +schema: "elementary"     +enabled: "{{ target.type != 'fabric' }}"
## Solution 2: Use vars to control Elementary Another approach is to use a variable to control Elementary's enablement, as suggested in the [Elementary Slack forum](https://slack.com/archives/C02CTC89LAX/p1669625539486519):
Copy code
yaml models:   elementary:     +schema: "elementary"     +enabled: "{{ var('enable_elementary', target.type != 'fabric') }}"
Then you can run your commands with:
Copy code
dbt build --vars '{"enable_elementary": false}' --target fabric
## Solution 3: Ensure package versions are compatible Make sure your Elementary package version is compatible with your dbt version. According to the [Elementary Slack forum](https://slack.com/archives/C02CTC89LAX/p1685079843099879), version mismatches can cause issues. ## Solution 4: Check for materialization conflicts The error message mentions materialization overrides. You've already added the required flag:
Copy code
yaml flags:   require_explicit_package_overrides_for_builtin_materializations: false
But make sure this is in the main project's dbt_project.yml as well, not just in the package. ## Additional troubleshooting steps: 1. Try running with 
--full-refresh
and 
--vars '{"elementary_full_refresh": true}'
as suggested in the [Elementary Slack forum](https://slack.com/archives/C02CTC89LAX/p1736511654372659). 2. Check if there are any materialization overrides in your project that might conflict with Elementary's requirements. 3. Make sure your database user has sufficient privileges to create the Elementary schema and tables. If these solutions don't resolve the issue, you might need to look more closely at how the Fabric adapter is handling boolean values in SQL, as the error suggests a problem with the 'True' value being treated as a column name.
c
So some more information: When I put the following code block directly into my repository, elementary correctly disables itself:
Copy code
models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.type != 'fabric' }}"
But when I put this same exact code block in an imported package in dbt_project.yml , I run into the same error again @kapa.ai
k
A message from kapa.ai
Thank you for providing that additional information. This helps clarify the issue you're experiencing with Elementary Data. The problem you're encountering is related to how dbt handles configuration inheritance between packages. When you define configuration in an imported package versus in your main project, there are different behaviors. Based on the knowledge sources, this is a known issue with disabling Elementary in imported packages. Several users in the Elementary Slack community have reported similar problems. The key insight is that variables and configurations defined in imported packages don't automatically get applied to the main project. According to the [Slack forum discussion](https://slack.com/archives/C02CTC89LAX/p1684508852749729), dbt compiles each package as an independent project, which is why your configuration in the imported package isn't working as expected. Here are some solutions: 1. Keep the configuration in your main project's dbt_project.yml (which you've confirmed works):
Copy code
yaml    models:      elementary:        +schema: "elementary"        +enabled: "{{ target.type != 'fabric' }}"
2. If you must configure it in the package, you need to provide a default value for any variables used:
Copy code
yaml    models:      elementary:        +schema: "elementary"        +enabled: "{{ var('enable_elementary', target.type != 'fabric') }}"
3. Alternative approach using vars: As mentioned in the [Slack forum](https://slack.com/archives/C02CTC89LAX/p1669625539486519), you can use vars to disable specific Elementary functionality:
Copy code
yaml    vars:      disable_run_results: "{{ target.type == 'fabric' }}"      disable_tests_results: "{{ target.type == 'fabric' }}"      disable_dbt_artifacts_autoupload: "{{ target.type == 'fabric' }}"      disable_dbt_invocation_autoupload: "{{ target.type == 'fabric' }}"
The most reliable solution appears to be keeping the Elementary configuration in your main project's dbt_project.yml file, which you've already confirmed works correctly.
c
I have to do this within the package. I cannot do in local as I'm developing the package. I set the error with the following config in dbt_project.yml in the package
Copy code
models:
  ## see docs: <https://docs.elementary-data.com/>
  elementary:
    ## elementary models will be created in the schema '<your_schema>_elementary'
    +schema: |
      {%- if var('tuva_schema_prefix',None) != None -%}{{var('tuva_schema_prefix')}}_elementary{% else %}elementary{%- endif -%}
    ## To disable elementary for dev, uncomment this:
    # enabled: "{{ target.name in ['prod','analytics'] }}"
    +enabled: "{{ var('enable_elementary', target.type != 'fabric') }}"

flags:
  require_explicit_package_overrides_for_builtin_materializations: false


vars:
    disable_run_results: "{{ target.type == 'fabric' }}"
    disable_tests_results: "{{ target.type == 'fabric' }}"
    disable_dbt_artifacts_autoupload: "{{ target.type == 'fabric' }}"
    disable_dbt_invocation_autoupload: "{{ target.type == 'fabric' }}"
    enable_elementary: "{{ target.type == 'fabric' }}"
Error:
Copy code
(.venv) chasejones@Chases-Mac-mini demo % dbt build --full-refresh -x
18:33:54  Running with dbt=1.8.6
18:33:54  Registered adapter: fabric=1.9.3
18:33:54  Unable to do partial parsing because saved manifest not found. Starting full parse.
18:34:02  Found 862 models, 97 seeds, 3 operations, 134 data tests, 1610 macros
18:34:02  
18:34:06  
18:34:06  Running 1 on-run-start hook
18:34:06  
IMPORTANT - Starting from dbt 1.8, users must explicitly allow packages to override materializations. 
Elementary requires this ability to support collection of samples and failed row count for dbt tests.
Please add the following flag to dbt_project.yml to allow it:

flags:
  require_explicit_package_overrides_for_builtin_materializations: false

Notes - 
* This is a temporary measure that will result in a deprecation warning, please ignore it for now. Elementary is working with the dbt-core team on a more permanent solution.
* This message can be muted by setting the 'mute_ensure_materialization_override' var to true.
        
18:34:06  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
18:34:06  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
18:34:06  
18:34:06  Concurrency: 4 threads (target='fabric')
18:34:06  
18:34:06  1 of 1045 START sql incremental model dbt_cjones.data_monitoring_metrics ....... [RUN]
18:34:06  2 of 1045 START sql incremental model dbt_cjones.dbt_columns ................... [RUN]
18:34:06  3 of 1045 START sql incremental model dbt_cjones.dbt_exposures ................. [RUN]
18:34:06  4 of 1045 START sql incremental model dbt_cjones.dbt_invocations ............... [RUN]
18:34:06  4 of 1045 ERROR creating sql incremental model dbt_cjones.dbt_invocations ...... [ERROR in 0.40s]
18:34:06  5 of 1045 START sql incremental model dbt_cjones.dbt_metrics ................... [RUN]
18:34:06  CANCEL query model.elementary.data_monitoring_metrics .......................... [CANCEL]
18:34:06  CANCEL query model.elementary.dbt_columns ...................................... [CANCEL]
18:34:06  CANCEL query model.elementary.dbt_exposures .................................... [CANCEL]
18:34:06  CANCEL query model.elementary.dbt_metrics ...................................... [CANCEL]
18:34:09  5 of 1045 OK created sql incremental model dbt_cjones.dbt_metrics .............. [OK in 2.86s]
18:34:09  3 of 1045 OK created sql incremental model dbt_cjones.dbt_exposures ............ [OK in 3.25s]
18:34:11  1 of 1045 OK created sql incremental model dbt_cjones.data_monitoring_metrics .. [OK in 5.49s]
18:34:18  2 of 1045 ERROR creating sql incremental model dbt_cjones.dbt_columns .......... [ERROR in 12.59s]
18:34:18  
18:34:18    Database Error in model dbt_invocations (models/edr/dbt_artifacts/dbt_invocations.sql)
  ('42S22', "[42S22] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid column name 'True'. (207) (SQLExecDirectW)")
  compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_invocations.sql
18:34:18  
18:34:18  Running 2 on-run-end hooks
18:34:20  Database error while running on-run-end
18:34:20  
18:34:20  Finished running 15 incremental models, 632 table models, 167 view models, 48 ephemeral models, 134 data tests, 97 seeds, 1 project hook in 0 hours 0 minutes and 18.01 seconds (18.01s).
18:34:21  
18:34:21  Completed with 1094 errors and 0 warnings:
18:34:21  
18:34:21    Database Error in model dbt_invocations (models/edr/dbt_artifacts/dbt_invocations.sql)
  ('42S22', "[42S22] [Microsoft][ODBC Driver 18 for SQL Server][SQL Server]Invalid column name 'True'. (207) (SQLExecDirectW)")
  compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_invocations.sql
@kapa.ai
k
A message from kapa.ai
New message text here
n
Hey @Chase Jones! MS fabric is not yet supported in Elementary, but luckily for you we’re working on it now thanks to a great contributor. I’ll give feedback to Elementary AI so it keeps improving with more accurate answers 🙌🏼
c
Thanks @Noya Offer. I'm trying to disable it for making it a dependency to my dbt package, but I cannot get it to disable if target = fabric. Any advice here, I'm at the point where I might need to fork it to get it to work as I'd like.
I want to create elementary as a dependency of my package, but only enable it where it's supported
7 Views
Open in Slack
PreviousNext
Powered by