https://linen.dev logo
Join Slack
Powered by
Hello <@U088WCP3B4P> ! We try to set up elementary...
# ask-elementary-ai
m
Hello @kapa.ai ! We try to set up elementary models creation. I have followed the installation instructions (https://docs.elementary-data.com/oss/quickstart/quickstart-cli-package): Addeed elementary to 
packages.yml
and addeed elementary to my 
dbt_project.yml
. After running the command: dbt run --select elementary I get the following error for the views creation, here is the example for model snapshot_run_results.sql. Logs:
Copy code
[0m15:36:55.189828 [error] [Thread-3 (]: [31mUnhandled error while executing target\run\elementary\models\edr\run_results\snapshot_run_results.sql[0m
'Principal'
[0m15:36:55.405611 [debug] [Thread-4 (]: SQL status: OK in 0.510 seconds
[0m15:36:55.582505 [debug] [Thread-3 (]: Traceback (most recent call last):
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\task\base.py", line 361, in safe_run
    result = self.compile_and_execute(manifest, ctx)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\task\base.py", line 308, in compile_and_execute
    result = self.run(ctx.node, manifest)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\task\base.py", line 408, in run
    return self.execute(compiled_node, manifest)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\task\run.py", line 334, in execute
    return self._execute_model(hook_ctx, context_config, model, context, materialization_macro)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\task\run.py", line 294, in _execute_model
    result = MacroGenerator(
        materialization_macro, context, stack=context["context_macro_stack"]
    )()
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\clients\jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt_common\clients\jinja.py", line 395, in call_macro
    return macro(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 770, in __call__
    return self._invoke(arguments, autoescape)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 784, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 186, in macro
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\sandbox.py", line 401, in call
    return __context.call(__obj, *args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 303, in call
    return __obj(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\clients\jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt_common\clients\jinja.py", line 395, in call_macro
    return macro(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 770, in __call__
    return self._invoke(arguments, autoescape)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 784, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 25, in macro
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\sandbox.py", line 401, in call
    return __context.call(__obj, *args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 303, in call
    return __obj(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\clients\jinja.py", line 82, in __call__
    return self.call_macro(*args, **kwargs)
           ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt_common\clients\jinja.py", line 395, in call_macro
    return macro(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 770, in __call__
    return self._invoke(arguments, autoescape)
           ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 784, in _invoke
    rv = self._func(*arguments)
  File "<template>", line 48, in macro
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\sandbox.py", line 401, in call
    return __context.call(__obj, *args, **kwargs)
           ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\jinja2\runtime.py", line 303, in call
    return __obj(*args, **kwargs)
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\dbt\adapters\spark\impl.py", line 503, in standardize_grants_dict
    grantee = row["Principal"]
              ~~~^^^^^^^^^^^^^
  File "C:\Users\magdalena.horosz\VirtualEnv\my_env\Lib\site-packages\agate\mapped_sequence.py", line 88, in __getitem__
    return self.dict()[key]
           ~~~~~~~~~~~^^^^^
KeyError: 'Principal'
k
A message from kapa.ai
Thank you for providing the context and following the installation instructions. Based on your description, you are encountering errors when running 
dbt run --select elementary
, specifically during the creation of views such as 
snapshot_run_results.sql
. From the knowledge sources, there are a few common causes and troubleshooting steps for issues related to view creation in Elementary: 1. Database/Adapter Limitations: Some data warehouses or connectors (such as certain Trino connectors) do not support view creation. If your underlying database does not support views, you will encounter errors like the one you described. In such cases, a workaround is to use a different catalog/connector that supports views, or try disabling view creation for Elementary models by setting 
+views_enabled: false
under the 
elementary
model configuration in your 
dbt_project.yml
. However, this is not officially documented and may cause unexpected behavior. If possible, point Elementary to a database/catalog that supports views, as some users have done with the Iceberg connector for Trino [Trino view creation issue](https://slack.com/archives/C02CTC89LAX/p1716561651807899). 2. Materialization Configuration: If you have a global materialization configuration in your 
dbt_project.yml
(e.g., setting all models to be materialized as views), this can override Elementary’s required materializations. Elementary needs some models to be incremental tables. Make sure your 
dbt_project.yml
is configured like this:
Copy code
yaml    models:      my_project:        materialized: "{{ 'table' if target.name == 'prod-cloud' else 'view' }}"      elementary:        +schema: "elementary"
If you change materialization settings, run:
Copy code
dbt run --select elementary --full-refresh
[Materialization config reference](https://docs.elementary-data.com/oss/quickstart/quickstart-cli-package#step-by-step%3A-install-elementary-dbt-package) 3. Adapter Support: Some adapters (e.g., dbt-spark) are not fully supported by Elementary, and may cause issues with model creation, especially for views. If you are using Databricks, make sure you are using the 
dbt-databricks
adapter, not 
dbt-spark
[Adapter support discussion](https://slack.com/archives/C02CTC89LAX/p1663248371176369). 4. Qualified Relation Names: For Redshift, you may see errors about relation names needing to be qualified when creating views with no schema binding. This was a known issue and has been fixed in recent releases. Make sure you are using the latest version of the Elementary dbt package [Redshift view creation issue](https://slack.com/archives/C02CTC89LAX/p1686685412667519). Next Steps: - Double-check your 
dbt_project.yml
for any global materialization settings. - Ensure your data warehouse/connector supports view creation. - If using Trino or a similar connector, consider switching to a catalog that supports views, or try the 
+views_enabled: false
workaround (with caution). - Make sure you are using the latest version of the Elementary dbt package. If you continue to encounter issues, please provide the specific error message from your logs and details about your data warehouse and dbt adapter, so I can assist you further.
A message from kapa.ai
Thank you for providing the detailed error log. Based on the traceback and the error message:
Copy code
KeyError: 'Principal'
This error occurs during the execution of a dbt model (specifically, when creating a view for 
snapshot_run_results.sql
) and is related to the Spark adapter (
dbt.adapters.spark.impl.py
). The error is triggered when the code tries to access a key named 
'Principal'
in a dictionary, but that key does not exist in the returned row. ### What does this mean? - This is a known issue when using the dbt-spark adapter with Elementary. The Elementary team has stated that, as of now, 
dbt-spark
is not an officially supported adapter for Elementary, and such errors can occur due to incompatibilities in how grants or permissions are handled in Spark versus other warehouses like Databricks or BigQuery. - The error is not directly related to your configuration of Elementary, but rather to the underlying support for Spark in the dbt adapter and how Elementary interacts with it. ### What can you do? - Switch to dbt-databricks if possible: If you are using Databricks, it is recommended to use the 
dbt-databricks
adapter instead of 
dbt-spark
, as Elementary officially supports Databricks and not Spark. This has been confirmed in several support threads and GitHub issues, where users running into similar issues were advised to switch adapters for compatibility reasons [dbt-spark not supported](https://github.com/elementary-data/elementary/issues/1441). - No official fix for dbt-spark: If you must use Spark, be aware that Elementary does not actively maintain this integration, and you may encounter further issues. The team welcomes community contributions for Spark support, but there is no out-of-the-box solution at this time. - Check your dbt version and adapter: If you are using Databricks, ensure you are using the correct adapter (
dbt-databricks
) and not 
dbt-spark
. You can check this by running:
Copy code
pip freeze | grep dbt-
and confirming the installed adapter. ### Summary Your error is due to the use of the 
dbt-spark
adapter, which is not officially supported by Elementary. The recommended solution is to use the 
dbt-databricks
adapter if possible. If you need to use Spark, you may encounter further issues, and there is currently no official fix from the Elementary team for this adapter [dbt-spark not supported](https://github.com/elementary-data/elementary/issues/1441). If you need further help or want to contribute a fix, the Elementary team encourages community contributions for Spark support.
m
@kapa.ai: Thanks you information! But the adapter, which I am using is: Registered adapter: databricks=1.10.0
k
A message from kapa.ai
Thank you for clarifying that you are using the 
dbt-databricks
adapter (version 1.10.0). Given this, the error you are seeing:
Copy code
KeyError: 'Principal'
is unexpected, as this issue is more commonly associated with the 
dbt-spark
adapter. However, there are a few relevant points from the knowledge sources that may help: 1. Adapter Confusion and Permissions Even when using the 
dbt-databricks
adapter, issues can arise if there are permission or grant-related misconfigurations. The error is triggered when dbt tries to standardize grants and expects a 
'Principal'
field in the response, which may not be present depending on your Databricks workspace setup or the way permissions are managed. This is not a common Elementary-specific error, but rather a dbt/Databricks integration issue. Similar issues have been reported as general Databricks-dbt connection problems, not specific to Elementary itself [see: connection issues](https://github.com/elementary-data/elementary/issues/1308). 2. Elementary and Databricks Compatibility Elementary officially supports Databricks via the 
dbt-databricks
adapter, and most users are able to run 
dbt run --select elementary
successfully if their Databricks and dbt profiles are configured correctly. If you encounter errors related to grants or permissions, it may be due to missing or misconfigured permissions for the service principal or user running the dbt job. Make sure the user or service principal has the necessary permissions on the schema and tables Elementary is trying to create or modify [see: Databricks permissions](https://docs.elementary-data.com/cloud/integrations/dwh/databricks). 3. Potential Workarounds - Double-check your Databricks workspace permissions, especially for the schema where Elementary models are being created. - If you have recently switched adapters or upgraded dbt/Databricks, try running a full refresh:
Copy code
dbt run --select elementary --full-refresh
- If the error persists, try dropping the problematic tables or schema manually and re-running the command, as sometimes old metadata can cause issues [see: manual table drop suggestion](https://slack.com/archives/C02CTC89LAX/p1672243967527919). 4. If the Issue Persists If none of the above resolves the issue, it may be a more recent or edge-case bug in the interaction between dbt-databricks and Elementary. In such cases, the best next step is to open an issue with detailed logs and your environment information, as the Elementary team actively supports Databricks and may be able to provide a fix or workaround [see: support channels](https://docs.elementary-data.com/cloud/integrations/dwh/databricks). Summary: You are using the correct adapter, but the error likely relates to Databricks permissions or a rare dbt-databricks integration issue. Double-check your permissions, try a full refresh, and consider manually cleaning up old tables. If the problem continues, reaching out with detailed logs to the Elementary team is recommended, as this is not a common or documented Elementary-specific error for the supported Databricks setup. If you can share more details about your 
profiles.yml
and 
dbt_project.yml
(with sensitive info redacted), I can help review your configuration for potential issues.
m
@kapa.ai: My dbt_project.yml looks the following:
Copy code
models:
  +grants:
    select: ["admin"]

  elementary:
    +schema: "_elementary"

flags:
  require_explicit_package_overrides_for_builtin_materializations: False
  source_freshness_run_project_hooks: True
The issue is caused by grating the "select" access to "admin" user specified at model level, but the user running the dbt job actually has the right permission to grant this access to elementary schema
k
A message from kapa.ai
Thank you for sharing your 
dbt_project.yml
and clarifying the context. Based on your configuration:
Copy code
yaml models:   +grants:     select: ["admin"]    elementary:     +schema: "_elementary"
and your observation that the error is caused by granting 
select
access to the 
"admin"
user at the model level, here’s what can be concluded from the knowledge sources: --- ### Key Points from the Knowledge Sources 1. Grants and Permissions in dbt/Elementary - Granting privileges at the model level using dbt's 
+grants
config is supported, but the actual ability to grant depends on the permissions of the user running the dbt job and the underlying data warehouse's requirements. - If the user running the dbt job has the correct permissions to grant access, but the error persists, it may be due to how the target warehouse (Databricks, in your case) interprets the grantee name or expects the grants to be structured. 2. Known Issues with Grants and Databricks - There have been issues reported where the expected field (like 
'Principal'
) is missing in the grants dictionary, leading to a 
KeyError
. This can happen if the grantee (in your case, 
"admin"
) does not exist as a principal in Databricks, or if the grants structure returned by Databricks does not match what dbt expects. - The knowledge sources suggest that such errors are often related to the specifics of the warehouse's grants implementation, not to Elementary itself. For example, if the grantee does not exist or is not recognized as a valid principal, Databricks may not return the expected structure, causing dbt to fail with a 
KeyError
on 
'Principal'
[see: similar grant/permission issues](https://github.com/elementary-data/elementary/issues/1308). 3. Troubleshooting Steps - Double-check that 
"admin"
is a valid principal in your Databricks workspace and that it matches the expected format for your warehouse. - Try removing or commenting out the 
+grants
section temporarily to confirm that this is the source of the error. - If the error disappears, you may need to adjust the grantee name or consult your Databricks admin to ensure the 
"admin"
principal exists and is grantable. - There is no explicit mention in the knowledge sources of a workaround within Elementary for this specific grants issue, but similar issues have been resolved by correcting the grantee or permissions in the warehouse. --- ### Summary The error is caused by the 
+grants
configuration attempting to grant 
select
to 
"admin"
, but Databricks/dbt cannot find or recognize 
"admin"
as a valid principal, resulting in a 
KeyError: 'Principal'
. This is a warehouse-specific grants issue, not an Elementary bug. Double-check the existence and validity of the 
"admin"
principal in your Databricks environment, and try running without the grants to confirm the root cause. If the issue persists, you may need to adjust your grants configuration or consult your Databricks administrator for the correct principal name or permissions structure. If you need further help, please provide the exact error message you receive when the grants are applied, as this can help pinpoint whether it's a naming issue or a deeper permissions problem. --- If you need more detailed guidance, please let me know the exact error message you see when the grants are in place, and I can help further!
m
@kapa.ai I have check out and "admin" principal exists, so it's not the reason for the error. I have also tried removing the 
+grants
section temporarily to confirm that this is the source of the error. Once 
+grants
section is removed, the error doesn't happen anymore, but this grant must be set at project level also to models in _elementary schema.
k
A message from kapa.ai
Thank you for the detailed troubleshooting steps. Based on the knowledge sources, your approach—removing the 
+grants
section and observing that the error disappears—confirms that the issue is directly related to how dbt/Databricks handles the grants configuration for your models. The knowledge sources indicate that granting privileges at the model level using dbt's 
+grants
config is supported, and users have successfully applied it to Elementary models in Snowflake and other warehouses. However, there is no explicit documentation or example in the knowledge sources confirming that this works seamlessly with Databricks, nor is there a known workaround for this specific 
KeyError: 'Principal'
when the principal exists and permissions are correct. The closest related guidance is to ensure the grants configuration is correct and that the grantee exists, which you have already done. The sources also suggest that such issues may be due to how the warehouse returns grant information, which can differ between environments and may not always match dbt's expectations. Unfortunately, there is no further troubleshooting or configuration advice in the knowledge sources for this specific scenario with Databricks and the 
admin
principal. In summary: The knowledge sources do not contain enough information to provide a definitive solution for this grants-related error with Databricks and Elementary. You have already followed the recommended troubleshooting steps. If the issue persists, it may require further investigation with Databricks or dbt support, as it appears to be an edge case not covered in the available documentation or community discussions. If you need to ensure grants are applied, you might consider applying them manually as a temporary workaround until a more robust solution is documented by the community or maintainers. If you have further details or logs, or if the issue is resolved in a future release, sharing that with the Elementary community may help others facing the same problem.
m
@kapa.ai is it possible to skip adding the grants (which are set at the model level in dbt_project.yml file) to elementary models?
k
A message from kapa.ai
Based on the knowledge sources, it is possible to skip adding grants to Elementary models by configuring your 
dbt_project.yml
so that the global 
+grants
setting does not apply to the 
elementary
models. You can do this by overriding or omitting the 
+grants
configuration specifically for the 
elementary
package. For example, you can explicitly set an empty grants configuration for Elementary models:
Copy code
yaml models:   +grants:     select: ["admin"]    elementary:     +schema: "_elementary"     +grants: {}
This approach ensures that the global grants do not propagate to the Elementary models, effectively skipping the grants for them. This method is supported in dbt and is referenced in discussions about controlling grants inheritance in dbt projects [dbt grants config inheritance](https://docs.getdbt.com/reference/resource-configs/grants#grant-config-inheritance). The knowledge sources do not mention any Elementary-specific limitations or issues with this approach, so it should work as expected. If you encounter any issues, you can further customize the grants configuration for subdirectories or individual models within the 
elementary
package as needed.
m
@kapa.ai unfortunately using "+grants: {}" for the elementary models, global grants are still propagated to the Elementary models. Is there any other workaround which can be applied?
k
A message from kapa.ai
I'm sorry, but the knowledge sources do not contain information about a reliable way to prevent global model-level 
+grants
from propagating to Elementary models in 
dbt_project.yml
. While overriding with 
+grants: {}
is a common dbt approach, there is no explicit confirmation in the sources that this works for Elementary, and you have observed that it does not prevent inheritance in your case. No alternative workarounds or configuration patterns for this specific grants inheritance issue with Elementary models are documented in the provided knowledge sources. If this behavior is critical for your workflow, you may need to consult dbt or Elementary support channels for further guidance, as this scenario is not addressed in the available documentation or community discussions.
37 Views
Open in Slack
PreviousNext
Powered by