Hi Team! I noticed something potentially dangerou...
# community-support
n
Hi Team! I noticed something potentially dangerous in
elementary-data
dbt package
0.24.0
. It defines a
get_incremental_microbatch_sql
macro that effectively disables batch execution, unless the
require_batched_execution_for_custom_microbatch_strategy
flag is explicitly set. That was pretty unexpected to discover it by full-refreshing a couple of internet-scale datasetsπŸ€¦β€β™‚οΈ
Some kind of post-mortem analysis - written by with Claude: 1. Since dbt 1.9, the microbatch strategy resolves through a macro named
get_incremental_microbatch_sql
. Macro lookup precedence is project β†’ installed packages β†’ dbt built-ins, so any package defining a root-level macro with that name shadows the built-in for the whole project. 2. dbt-core then checks who won the lookup (
Manifest.use_microbatch_batches
, dbt 1.11): return ( get_flags().require_batched_execution_for_custom_microbatch_strategy or self._microbatch_macro_is_core(project_name=project_name) )
_microbatch_macro_is_core
returns true only if the winning macro has
Locality.Core
(= built-in). This is a backwards-compat decision: before native microbatch existed, people implemented "microbatch" as a custom strategy via this exact macro-name convention, and those models ran single-pass β€” dbt won't silently change their behavior. 3. That boolean picks the runner (
dbt/task/run.py
):
MicrobatchModelRunner
(splits begin→now into day batches, provides
model.batch
, injects per-batch
event_time
filters on inputs) vs the ordinary incremental runner β€” one compile, one execution,
model.batch
is none, no time filters β†’ the model reads full source history in a single pass. 4. elementary 0.24.0 ships exactly such a macro β€” a wrapper to capture per-batch compiled SQL, added in https://github.com/elementary-data/dbt-data-reliability/pull/1000: https://github.com/elementary-data/dbt-data-reliability/blob/0.24.0/macros/edr/dbt_artifacts/microbatch/capture_microbatch_compiled_c ode.sql#L20 So just adding the package to
packages.yml
flips every microbatch model in the host project into single-pass mode. The only signal is a
MicrobatchMacroOutsideOfBatchesDeprecation
warning at parse time β€” easy to miss among other deprecations. The write side stays valid incremental DML (we verified no duplicates), the damage is read-side: full-history scans, ~10–15x runtime in our case.
o
Hi @Nikita Kotlyarov , thanks for surfacing- our team will look into it and follow up
❀️ 1
g
Hi @Nikita Kotlyarov, thanks for flagging this! In our new release (
0.25.0
) the macro was renamed, so now it won't be used by dbt when checking
use_microbatch_batches
, only if you explicitly declare an override in your own project.
thankyou 1