https://linen.dev logo
Join Slack
Powered by
<@U088WCP3B4P> what are the dbt dependencies of th...
# ask-elementary-ai
o
@kapa.ai what are the dbt dependencies of the 0.26.0 version? if i choose the elementary[databricks] version, does it pull the dbt-databricks as dependecy?
k
Thinking completed The provided knowledge sources do not explicitly detail the exact dependencies for the 0.26.0 release specifically, nor do they provide the complete dependency specification for the 
elementary-data[databricks]
extra. However, based on the available information across versions: 1. dbt dependencies: ā—¦ When installing the Elementary dbt package (
elementary-data/elementary
) in a dbt project, it pulls 
dbt-labs/dbt_utils
as an upstream dependency [Linen thread; Linen thread]. ā—¦ Elementary CLI (
edr
) relies internally on 
dbt-core
and uses an internal dbt project that also installs and runs 
elementary
and 
dbt-utils
[Linen thread; Slack post]. 1. Does 
elementary-data[databricks]
pull 
dbt-databricks
? ā—¦ The knowledge sources do not provide the explicit 
setup.py
/ 
pyproject.toml
definition for the 
[databricks]
extra. ā—¦ What is documented is that running 
pip install 'elementary-data[databricks]'
is the required command to install the CLI connector for Databricks [CLI install; Linen thread], and users typically manage 
dbt-databricks
alongside Elementary in Databricks environments [Linen thread; Linen thread]. For the exact dependency list and version constraints of 0.26.0, you can check the package definition directly on Libraries.io for elementary-data or the Elementary GitHub repository [Slack forum question]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
does the cli rely on dbt v2 version? It was announced that it is supporting it
@kapa.ai does the cli rely on dbt v2 version? It was announced that it is supporting it
k
Thinking completed The knowledge sources indicate that the Elementary CLI (
edr
) does not rely on dbt v2. Instead, it still relies on 
dbt-core
. What the knowledge sources clarify: ā€¢ dbt v2 / dbt Fusion support is for the dbt package: Starting with version 0.20, Elementary introduced beta support for dbt Fusion (dbt v2) specifically for the Elementary dbt package [dbt Fusion OSS]. ā€¢ The CLI does not use Fusion directly: The CLI operates separately by reading from the Elementary schema tables created in your warehouse. It uses an internal dbt project that depends on 
dbt-core
[Linen thread; Linen thread]. ā€¢ Packaging conflicts: Because the 
elementary-data
CLI package requires 
dbt-core
as a dependency, installing 
elementary-data
into the same virtual environment alongside 
dbt==2.0.0
(dbt Fusion) pulls in and defaults to 
dbt-core
rather than dbt Fusion [Linen thread; Slack forum user question]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai for project hooks and etc. the dbt package elementary is crucial with the elementary target right. The CLI is to send alerts and generate elementary report right? So technically the CLI is not needed for normal dbt runs, only after it to eg. send out alert
k
Thinking completed Yes, that is completely correct. Here is how the roles are divided based on the knowledge sources: ā€¢ Normal dbt runs (dbt package): You do not need the CLI installed or running for your standard dbt commands (
dbt run
, 
dbt test
, 
dbt build
) [Linen discussion; Linen thread]. As long as the Elementary dbt package is installed in your dbt project, its 
on-run-end
hooks automatically capture test results, run results, and metadata artifacts and write them directly into your data warehouse under the Elementary schema [Linen discussion; Elementary in production]. ā€¢ Elementary CLI (
edr
): The CLI is a completely separate Python tool used after or independently of your dbt runs [Linen thread]. Its primary purposes are to connect to your warehouse, read from the Elementary tables, and: a. Generate and distribute HTML reports (
edr report
, 
edr send-report
) [Elementary in production; Linen discussion]. b. Send alerts to destinations like Slack (
edr monitor
) [Elementary in production]. ā€¢ About the target / profile: ā—¦ For your dbt project runs, dbt runs against your normal environment target/profile (the package only requires read access to your schemas and write access to the Elementary schema) [Elementary in production]. ā—¦ The dedicated 
elementary
profile in 
profiles.yml
is specifically required by the Elementary CLI so that it knows how to authenticate and read data from your warehouse [Elementary in production; Linen discussion]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai i'm trying dbt 2.0. and we have this config in dbt_project.yml
Copy code
elementary:
    +schema: dbt_elementary
    +enabled: "{{ target.name not in  ['playground', 'ci', 'develop', 'dev_manifest'] }}"
and if i run dbt run with target develop the elementary hooks are triggered, if i go back to dbt v.1.12 then it doesnt happen
k
Thinking completed The knowledge sources do not explicitly explain why dbt 2.0 (dbt Fusion) behaves differently from dbt 1.12 regarding 
+enabled
evaluated on packages, but they do highlight a few key details about how Elementary hooks interact with project configuration. 1. +enabled: false Disables Models, Not Hooks In dbt, configuring 
+enabled
under 
models.elementary
disables the compilation and materialization of Elementary's models, but it does not inherently prevent package-level 
on-run-end
hooks from executing [Linen thread]. In older dbt-core versions, disabling all models in a package sometimes suppressed certain hook behaviors or appeared inactive, whereas the rewritten engine in dbt 2.0 (Fusion) strictly parses and executes all package hooks defined in 
dbt_project.yml
regardless of model enablement [dbt Fusion Beta]. 2. The Recommended Approach: Use Hook Disable Vars To stop Elementary's 
on-run-end
hooks from running queries or uploading artifacts in specific targets, Elementary officially recommends using its hook disablement variables rather than 
+enabled
[Limit Hooks to Production; Limit Hooks OSS]. Add the following to the 
vars
section of your `dbt_project.yml`:
Copy code
vars:
  disable_dbt_artifacts_autoupload: "{{ target.name in ['playground', 'ci', 'develop', 'dev_manifest'] }}"
  disable_run_results: "{{ target.name in ['playground', 'ci', 'develop', 'dev_manifest'] }}"
  disable_tests_results: "{{ target.name in ['playground', 'ci', 'develop', 'dev_manifest'] }}"
  disable_dbt_invocation_autoupload: "{{ target.name in ['playground', 'ci', 'develop', 'dev_manifest'] }}"
These variables instruct Elementary's hooks to skip collecting metadata, run results, test results, and invocations for those targets [Can I disable hooks; Limit Hooks to Production]. This method is the standard supported pattern across environments and also ensures that any Elementary tests you might want to run in CI/dev will still function without overhead [Limit Hooks to Production; Linen thread]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai ok i pasted those vars and ran dbt compile and still got this
Copy code
Finished [  0.26s] hook  elementary-on-run-start-0
  Finished [  0.11s] hook  elementary-on-run-end-0
k
Thinking completed This is the expected behavior. The configuration variables control what queries and actions Elementary actually executes inside the hook, rather than preventing dbt from calling the hook macro itself [Linen thread; Linen thread]. Here is what is happening: 1. dbt still invokes the hook: Because the Elementary package defines 
on-run-start
and 
on-run-end
hooks in its project configuration, dbt will always call the entry-point macro and display the 
hook elementary-on-run-...
log lines in your console output [Linen thread; Linen thread]. There is no setting to suppress those dbt log lines entirely [Linen thread]. 2. The work inside the hook is skipped: When those disable variables evaluate to 
true
, the hook code evaluates the conditions, skips compiling queries, skips metadata collection, and does not upload any artifacts or run results to your warehouse [On-run-end hooks; Linen thread]. As seen in your output:
Copy code
Finished [ 0.11s] hook elementary-on-run-end-0
Taking only a fraction of a second (0.11s) confirms that the hook recognized the disable flags, exited early, and performed no warehouse queries or artifact uploads. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai so this is a behavior change from v1 dbt?
k
Thinking completed The knowledge sources do not explicitly explain the internal engine differences between dbt Core (v1) and dbt Fusion (v2) regarding why your specific project didn't show those hooks in v1.12. However, based on the documented behavior across versions: 1. In dbt Core (v1), package hooks always execute: In dbt Core (v1), defining 
+enabled: false
on models only disables the package's models, not its hooks [Slack user question; Linen thread]. Even when all disable variables are set, dbt Core still triggers the hook macro and logs the hook execution lines in the console [Linen thread; Slack thread]. 2. What changed with dbt 2.0 (dbt Fusion): dbt Fusion is a complete rewrite of the dbt engine. While the knowledge sources do not detail the exact internal parsing changes between the engines, they note that dbt Fusion changed how flags and hook rendering are handled (such as requiring valid rendered SQL from hooks and altering behavior around project flags) [Linen thread]. In short: seeing the hook start/finish log lines run in a fraction of a second (without performing uploads) has always been the standard, documented behavior in dbt Core (v1) as well [Linen thread]. If dbt 1.12 completely suppressed those log lines under your previous configuration, that discrepancy is due to how dbt Fusion handles project/package hook resolution compared to the Python-based dbt-core runner. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai you're docs are saying this
Copy code
Disable the whole package (Elementary tests won't work) -
models:
  elementary:
    +schema: "elementary"
    +enabled: "{{ target.name in ['prod','analytics'] }}"
k
Thinking completed Yes, the documentation does provide that configuration snippet under the option to "Disable Entire Package" or disable Elementary for non-production targets [Reduce on-run-end time; Dev/Prod Configuration]. However, the knowledge sources highlight two critical details regarding what this actually does: 1. It only disables models and tests, not the hook registration itself: Because this configuration is placed under the 
models:
block in 
dbt_project.yml
, setting 
+enabled: false
tells dbt not to compile or build Elementary's models (and prevents Elementary tests from running) [Reduce on-run-end time; Slack post]. It does not remove the 
on-run-start
and 
on-run-end
hooks that Elementary registers at the package level [Linen thread; GitHub issue]. 2. The hooks will still appear in your logs: Even when using 
+enabled: false
, dbt will still invoke the hook entry point and output the 
hook elementary-on-run-start
and 
hook elementary-on-run-end
log lines [Linen thread; GitHub issue]. The hooks simply complete quickly (in milliseconds) without performing updates [Linen thread]. 3. Official recommendation: Because disabling the package with 
+enabled: false
breaks or skips Elementary tests entirely, the documentation recommends using the 
vars
approach instead [Reduce on-run-end time; Dev/Prod Configuration]vars disable_dbt_artifacts_autoupload: "{{ target.name not in ['prod', 'analytics'] }}" disable_run_results: "{{ target.name not in ['prod', 'analytics'] }}" disable_tests_results: "{{ target.name not in ['prod', 'analytics'] }}" disable_dbt_invocation_autoupload: "{{ target.name not in ['prod', 'analytics'] }}"This allows your tests to run without the artifact upload overhead, although dbt will still display the hook start/finish log lines in your console [Reduce on-run-end time; Linen thread]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
o
@kapa.ai in the v1.12. dbt-core version + databricks adapter, the hooks are not printed/indicated to be run/skipped during compile
Copy code
uv run dbt compile
18:50:23  Running with dbt=1.12.3
18:50:50  Registered adapter: databricks=1.12.5
18:50:50  Unable to do partial parsing because a project config has changed
18:51:05  Found 248 models, 1026 data tests, 19 seeds, 5 snapshots, 74 sources, 2408 macros, 30 unit tests
18:51:05
18:51:05  Concurrency: 10 threads (target='playground')
18:51:05
k
Thinking completed The knowledge sources do not explicitly document the exact dbt-core internal code differences between dbt 1.x and dbt 2.0 (Fusion) regarding why 
dbt compile
behaves differently with hooks. However, based on the provided knowledge sources, this difference comes down to how each runner handles the 
compile
task: 1. dbt-core (v1.x) during `dbt compile`: In standard dbt-core v1, 
dbt compile
compiles your project manifest, models, and tests, but it does not execute 
on-run-start
and 
on-run-end
project hooks. Those hooks are executed during runnable tasks like 
dbt run
, 
dbt test
, and 
dbt build
[Linen thread; Slack thread]. As a result, no hook start/end lines are executed or logged during 
compile
. 2. dbt 2.0 (dbt Fusion) during `dbt compile`: In dbt Fusion, the compilation engine actually renders and evaluates 
on-run-start
and 
on-run-end
hooks as part of the 
compile
task [Slack user question]. Because Fusion renders and checks the hook SQL during compilation, it outputs the hook execution/finishing logs:Finished [ 0.26s] hook elementary-on-run-start-0 Finished [ 0.11s] hook elementary-on-run-end-0 Because you configured the disablement variables, the hook simply rendered and exited immediately (taking only 0.11s) without uploading artifacts or executing warehouse queries [Linen thread; Slack user question]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @elementary.ai in the thread for follow-ups or DM the app for 1-to-1 help.
9 Views
Open in Slack
PreviousNext
Powered by