@Rocky Shared-data 4.0.12 to 4.1.3 upgrade:

Inconsistent bundle_file_offsets across txn logs

returns under real write load, after a fully quiesced roll Setup: shared-data mode, 3 FE + 3 CN on Kubernetes, S3 storage volume. Upgraded 4.0.12 to 4.1.3. Procedure followed 1. Paused all external writers. 2. Drained every database to

running = 0

(

SHOW PROC '/transactions/<DbId>'

). 3. Disabled compaction, tablet balancing and statistics collection. 4.

ALTER SYSTEM CREATE IMAGE

, verified present on all 3 FE. 5. Rolled CN to 4.1.3 first, then FE. Both clean, no metadata replay errors. What worked on 4.1.3, against data written by 4.0.12 • A compaction cycle on a primary-key table with

file_bundling = true

, published to VISIBLE. CompactVersion advanced and the compaction score dropped. • An explicit multi-statement transaction (BEGIN, two INSERTs, COMMIT) on a

file_bundling = true

primary-key table, published to VISIBLE. One txn id, two load ids. I also confirmed 4.1.3 contains the

segment_metas

normalizer from #74107 and its backport #74143, and that

lake_proto_normalizer

is wired into the TabletManager load and save paths. What failed About ten minutes after resuming normal writers, one partition wedged:

Fail to publish partition <p> of txn <t>: Fail to publish version for tablets:[[...]],
error msg: Inconsistent bundle_file_offsets across txn logs for tablet <x>:
some logs have offsets, some don't. Cannot safely merge rowsets.

One transaction, 8 tablets, retried roughly once per second and never publishing. The affected table is a primary-key table with

file_bundling = true

, written by a high-frequency sync job that had a backlog to move. What did not help

lake_enable_batch_publish_version = false

. As I read it, that batches multiple versions, while this merge happens within a single transaction, so it never reaches the guard. My questions 1. Because every database was drained to

running = 0

before the roll, this transaction's logs should all have been written by 4.1.3. If that is correct, then "some logs have offsets, some don't" is not old-format versus new-format data. Can 4.1.3 itself produce one transaction whose rowsets bundle inconsistently, for example when write sizes vary within the transaction? Or is there a path where a pre-upgrade log can still enter this merge despite the drain? 2. Is

enable_file_bundling = false

a valid mitigation here? Does it need a restart, or does the mutable setting take effect for new writes? 3. Is there a supported way to unstick an already-committed transaction in this state, short of downgrading? 4. Is this a known issue, and is a fix targeted at 4.1.4? I still have the cluster in the failed state and can pull further diagnostics.

This error,

Inconsistent bundle_file_offsets across txn logs

, is a specific safety check in StarRocks 4.1.x shared-data (lake) mode. It occurs when a single transaction attempts to publish multiple rowsets (txn logs) for the same tablet, but those rowsets have inconsistent metadata regarding the

file_bundling

optimization (i.e., some logs contain bundled segment offsets while others do not). 1. Can 4.1.3 produce inconsistent bundling in a single transaction? Yes. Even if the upgrade was fully quiesced and all logs were written by 4.1.3, a single transaction can still produce inconsistent logs if: * Mixed Write Content: A transaction contains both data rowsets (which are bundled) and delete-only rowsets (which do not have segment files to bundle). * Write Size Variation: Within a high-frequency sync job, some writes might be large enough to trigger bundling logic while others are too small, leading to mixed metadata within the same transaction. * Multiple Senders: In multi-sender transactions (like multi-table stream loads or multi-statement SQL), different CN nodes might handle different rowsets for the same tablet. If there is a race or configuration lag in picking up the

file_bundling

property, they may write rowsets inconsistently. 2. Is

enable_file_bundling = false

a valid mitigation? Yes. This is the recommended mitigation to prevent further occurrences. * Mutable Setting: The

file_bundling

property is a mutable table/partition property. You can disable it using:

sql
    ALTER TABLE <table_name> SET ("file_bundling" = "false");

* Effect: New writes will immediately stop using bundling, and the CN will no longer attempt to generate bundled rowsets. This does not require a restart. 3. How to unstick an already-committed transaction? Since the transaction is already committed in the FE but failing to publish in the CN, the partition is effectively wedged because version advancement is blocked. * Try Disabling Bundling: Set

file_bundling = false

on the table. In some cases, the publish logic on the CN will re-evaluate the merge requirement. If bundling is disabled, it may bypass the strict consistency check and allow the publish to proceed (treating the existing offsets as raw data or normalizing them). * Check

lake_enable_batch_publish_version

: Although you mentioned this didn't help, ensuring it is

false

is correct for debugging, as it avoids batching multiple different transactions. * Manual Intervention: If disabling the property doesn't unstick it, the only supported way to "clear" it without a downgrade is often to wait for a patch (4.1.4) or, if the data can be reloaded, to truncate and reload. There is no simple

CANCEL

command for a transaction that has already reached the

COMMITTED

state. 4. Is this a known issue for 4.1.4? Yes, this is a known edge case in the 4.1.x line where the

lake_proto_normalizer

introduced in #74107 is sometimes too strict during the merge of rowsets within a single transaction. * Target Fix: A fix is targeted for 4.1.4 which relaxes this check, allowing the CN to safely merge bundled and non-bundled rowsets by normalizing the metadata (filling in dummy offsets for non-bundled segments) during the publish/merge phase. Recommendation: 1. Disable

file_bundling

on the affected table immediately to stop the "wedging" of new partitions. 2. If the transaction remains stuck after setting

file_bundling = false

, you may need to escalate to a 4.1.4 patch or a hotfix that includes the relaxed merge check. 3. You can verify the transaction state using

SHOW PROC '/transactions/<DbId>'

to see if the

Timeout

is approaching, though committed transactions will retry indefinitely until publish succeeds. References • data_source/feature-support-data-lake-analytics.md