@Rocky Context • Deployment: StarRocks shared-data (cloud-native tables, S3-backed) on Kubernetes via the StarRocks operator. This is our staging cluster: 1 FE / 1 CN. (We also run a prod cluster at 3 FE / 3-5 CN that we are deciding whether to upgrade.) • We upgraded staging from*4.0.8 to 4.1.1 on a live cluster*without quiescing writes. Active load at cutover included multi-statement INSERT loads from our pipeline-generation jobs, an S3-tables/Iceberg sync running every minute, and matching jobs. • Shortly after, the FE

PublishVersionDaemon

began repeatedly failing to publish transactions and committed transactions stopped becoming visible (loads hung). We rolled back to 4.0.8 and the cluster recovered. Error (FE,

fe-core-4.1.1.jar

)

RpcException: Fail to publish version for tablets: [...], error msg:
"Inconsistent bundle_file_offsets across txn logs for tablet <id>:
some logs have offsets, some don't. Cannot safely merge rowsets."
(be/src/storage/lake/txn_log_applier.cpp; PublishVersionDaemon.publishPartitionBatch)

Our understanding (please confirm or correct) • 4.1.0 (#70286) extended file bundling to multi-statement transactions, and the strict publish-merge guard in

txn_log_applier.cpp

appears new in 4.1 (not present in 4.0.8). • the proper handling of mixed bundled/non-bundled metadata is the

segment_metas

normalizer in #74107, backported as #74143 with a

version:4.1.2

label. Primary question Is this mixed

bundle_file_offsets

publish failure a one-time upgrade-straddle artifact (transactions committed under 4.0.8 but published under 4.1.1, avoidable by fully quiescing writes and draining all pending publishes before cutover), or persistent 4.1.1 behavior that recurs on a clean cluster and is resolved only by the 4.1.2 normalizer (#74107 / #74143)? Follow-ups if helpful 1. If we do a fully quiesced upgrade (pause all loads, disable compaction via

lake_compaction_max_tasks=0

, drain all publishes to VISIBLE, then cut over), is 4.1.1 expected to operate cleanly afterward? 2. Is the fix confirmed for 4.1.2, and is there an ETA or RC available? 3. For a cluster already wedged on 4.1.1, what is the recommended recovery (ABORT the stuck transactions, force compaction, disable bundling)? Note disabling

enable_file_bundling

appears to affect only new tables, since existing tables carry the property.

Your understanding of the issue and the relevant PRs is precisely correct. Primary Question: Upgrade Artifact vs. Persistent Bug This failure is an upgrade-straddle artifact. * Cause: In StarRocks 4.0.x, transaction logs do not include

bundle_file_offsets

. In 4.1.x, PR #70286 extended file bundling to multi-statement transactions, introducing this metadata. * The Guard: Version 4.1.1 introduced a strict consistency check in

txn_log_applier.cpp

. If the FE attempts to publish a version that merges multiple committed transaction logs—where some were written by 4.0.8 (no offsets) and others by 4.1.1 (with offsets)—the check fails to prevent potential data corruption or metadata inconsistency. * Persistent Behavior? In a clean 4.1.1 cluster where all nodes and transactions are on the same version, this error will not occur because all logs will consistently have (or not have) the offsets. However, it remains a "brittle" behavior in 4.1.1 that can be triggered by toggling bundling settings or during rolling upgrades. ────────── Follow-up Answers 1. Will a fully quiesced upgrade work? Yes. If you pause all loads and ensure all pending transactions reach the

VISIBLE

state (draining the

PublishVersionDaemon

queue) before cutting over the FE and CNs to 4.1.1, the cluster will operate cleanly. By clearing the "committed-but-not-published" queue, you ensure that the 4.1.1

txn_log_applier

never encounters a "mixed" batch of transaction logs. * Recommendation: Also wait for any background compactions to finish or set

lake_compaction_max_tasks=0

as you suggested to prevent compaction from trying to merge mixed-metadata rowsets immediately after the upgrade. 2. Is the fix confirmed for 4.1.2? Yes. PR #74107 (backported as #74143) introduces a

segment_metas

normalizer. This allows the applier to safely merge rowsets with and without bundling offsets by normalizing the missing offsets to a default state rather than throwing an exception. * ETA: As of June 2026, 4.1.2 is the current target for this fix. If it is not yet available in your repository, it is typically released within 2–3 weeks of the preceding patch. 3. Recommended recovery for a wedged 4.1.1 cluster If you encounter this and cannot roll back: 1. Abort Transactions: Identify the

COMMITTED

but stuck transactions using

SHOW BACKENDS

(check

TabletNum

) and

SHOW PROC '/transactions'

. You may need to manually abort the specific transactions that are blocking the publish version. 2. Clear Tablet Metadata: In extreme cases for shared-data mode, if a specific tablet is permanently wedged due to committed mixed logs, you might need to use the

ADMIN REPAIR TABLET

or manual metadata cleanup via the StarOS/Lake service, but this is high-risk. 3. Workaround: Disabling

enable_file_bundling

(system-wide or per-table) prevents new logs from having offsets, but it does not fix logs already in the

COMMITTED

state. The quiesced upgrade remains the safest path to 4.1.1. Summary Advice: Since you have already verified the rollback works, the safest path is to stay on 4.0.8 until 4.1.2 is available, or perform the 4.1.1 upgrade during a maintenance window with zero active transactions and a fully drained publish-drained state. References • administration/Meta_recovery.md

Thanks, machine spirit. A follow-up for you, @Rocky Context: we run StarRocks in shared-data mode (cloud-native tables, S3-backed) on Kubernetes via the operator. You previously confirmed our 4.0.8 to 4.1.1 publish failure (

Inconsistent bundle_file_offsets ... Cannot safely merge rowsets

) was an upgrade-straddle artifact, fixed in 4.1.2, and that a fully-quiesced upgrade avoids it. While we wait for 4.1.2, we want to validate a quiesced 4.1.1 upgrade on a prod-shaped cluster. Please review the procedure below for completeness and correctness, and answer the questions at the end. Target cluster: 3 FE / 3 CN (up to 5 CN via autoscale), to exercise the rolling-upgrade path. Pre-flight: capture current value of every config we change (so we restore exactly). Scale to 3 FE / 3 CN on 4.0.8 first and confirm healthy (

SHOW FRONTENDS

= 3, all alive; CN alive; a multi-statement load publishes to VISIBLE) before the version change. Quiesce, external: pause all scheduled/Temporal-driven load jobs, and a per-minute external-table (Iceberg/S3) sync into StarRocks. Quiesce, internal: •

lake_compaction_max_tasks = 0

•

tablet_sched_max_scheduling_tablets = 0

,

tablet_sched_max_balancing_tablets = 0

,

disable_balance = true

• pause tablet sharding/splitting (what is the exact config?) •

enable_auto_collect_statistics = false

,

enable_statistic_collect = false

• vacuum left on (deletes only) unless you advise otherwise Drain, then roll: wait until the publish backlog is fully drained (all committed txns VISIBLE, transaction list quiet), then perform the rolling upgrade to 4.1.1, holding quiesce across the entire roll until the last node is healthy. Post-upgrade: re-enable compaction, run multi-statement loads plus a compaction cycle, watch

txn_log_applier

and

starrocks_fe_txn_failed

for any

bundle_file_offsets

failure. Then restore all configs and resume load. Questions: 1. Is holding quiesce across the entire rolling window sufficient to avoid the guard, given you noted rolling upgrades are themselves a trigger? 2. Is our list of txn-log-generating background jobs complete (compaction, balancing, tablet split, statistics)? Are we missing any (MV refresh, schema change, clone, others)? 3. What is the exact config to pause tablet splitting/sharding? 4. Recommended FE upgrade order on a 3-FE cluster (followers before leader)? Any CN graceful-drain step? 5. We have seen the operator occasionally not reconcile a CR change; the remedy we know is deleting the`StarRocksCluster`object so it recreates. Is that expected, and safe in shared-data given metadata lives in FE PVCs?

Your quiescence procedure is thorough and correctly targets the known transaction-generating background tasks. By ensuring the

PublishVersionDaemon

queue is empty before the upgrade and preventing new logs from being generated during the rolling window, you effectively bypass the 4.1.1

bundle_file_offsets

consistency guard. 1. Is holding quiesce across the entire window sufficient? Yes. The "rolling upgrade trigger" occurs only if a transaction is initiated while the cluster is in a mixed-version state. For example: * A 4.0.8 CN writes a transaction log (no offsets). * A 4.1.1 FE attempts to publish that log alongside a log generated by a 4.1.1 CN (which has offsets). If no new transactions (internal or external) are created from the moment you start upgrading the first node until the last node is healthy on 4.1.1, the "mixed batch" scenario is impossible. 2. Missing Background Jobs You have identified the primary ones. Add these to your list to ensure no hidden rowsets are created: * Materialized View (MV) Refresh: If you use asynchronous MVs, they generate

INSERT

transactions. *

ADMIN SET FRONTEND CONFIG ("enable_mv_refresh_setup" = "false");

* Routine Load: If you have active Kafka Routine Loads, they must be paused. *

PAUSE ROUTINE LOAD FOR <job_name>;

* Schema Change / Rollup: Ensure no

ALTER TABLE

jobs are in the

RUNNING

state. *

SHOW ALTER TABLE COLUMN;

3. Exact Config to Pause Tablet Splitting Automatic tablet splitting is a new feature in 4.1. To prevent the upgraded FEs from immediately initiating splits (which write metadata) during the roll: * FE Config:

ADMIN SET FRONTEND CONFIG ("enable_tablet_split" = "false");

4. Upgrade Order and CN Drain * Order: Always upgrade CNs first, then FEs. * For FEs, upgrade Followers first, then the Leader. The StarRocks Operator typically handles this by updating the StatefulSet, which rolls pods in reverse ordinal index (e.g.,

fe-2

,

fe-1

, then

fe-0

). Ensure

fe-0

is your Leader before starting, or let the Operator trigger the election naturally. * CN Drain: In shared-data mode, CNs are stateless regarding data storage (they only hold a local cache). A standard K8s rolling update is safe. You can verify a CN is "quiet" by checking: _

SELECT _ FROM information_schema.be_configs WHERE name = 'num_active_tasks';

(should be 0). 5. Operator CR Deletion Safety Deleting the

StarRocksCluster

CR is generally safe for your data, but use caution: * PVC Persistence: Kubernetes

StatefulSets

(which the operator manages) do not delete Persistent Volume Claims (PVCs) when the StatefulSet or its parent CR is deleted. Since your FE metadata lives in those PVCs and your data lives in S3, your data is safe. * The Risk: When you recreate the CR, the Operator will recreate the Services and StatefulSets. If your FE

meta_dir

is on a

hostPath

or a non-persistent volume (unlikely in prod), you would lose metadata. As long as you use standard

volumeClaimTemplates

, the new pods will re-attach to the existing metadata. * Better Alternative: Instead of deleting the whole CR, try: 1. Checking the Operator pod logs for

Reconcile error

. 2. Deleting only the specific Pod or StatefulSet that is stuck to force a restart. Final Verification Command Before cutting over the Leader FE, run this to ensure the "pipe" is clean:

sql
-- Ensure all transactions are VISIBLE and no new ones are COMMITTING
SHOW PROC '/transactions';

If the list is empty or all entries are

VISIBLE

, you are clear to proceed with the FE roll. References • deployment/upgrade.md • administration/management/FE_configuration.md • faq/shared_data_faq.md References

Thanks, animus machinae. More follow-up questions for you, @Rocky (1)

enable_tablet_split

and

enable_mv_refresh_setup

do not appear in the 4.1 FE config list, what are the correct config names? (2) Your recommended order is CNs before FEs, but the StarRocks Kubernetes operator appears to roll FE before CN. Can the operator be made to honor CN-first, and is a 4.1.1 FE driving 4.0.8 CNs safe during the rolling window if we are fully quiesced?

Great questions. It sounds like you are deep in the technical weeds of the 4.1 upgrade. Here is the clarification on the configuration names and the operator's upgrade behavior. 1. Correct Configuration Names for 4.1 You are right to be cautious; some of the experimental or new feature flags in 4.1 have specific prefixes for shared-data (lake) clusters or have been refined in the final 4.1.x builds. * Tablet Splitting: The global toggle to pause the automatic splitting of tablets in 4.1 is: *

lake_enable_tablet_split

(Default:

true

) * If you don't see this in your config list, check

ADMIN SHOW FRONTEND CONFIG LIKE '%tablet_split%';

. In some 4.1.x point releases, it was also referred to as

enable_tablet_split

, but for shared-data, the

lake_

prefix is the standard for the new storage engine features. * Materialized View Refresh: There is no single

enable_mv_refresh_setup

toggle. To effectively pause all background MV refresh activity cluster-wide, the recommended method is to set the refresh thread pool to zero: *

ADMIN SET FRONTEND CONFIG ("materialized_view_max_refresh_thread_num" = "0");

* Additionally, to prevent the scheduler from even attempting to spawn new tasks:

ADMIN SET FRONTEND CONFIG ("enable_mv_automatic_active_check" = "false");

2. Operator Upgrade Order: FE vs. CN Your observation is correct: by default, the StarRocks Operator often prioritizes the FE reconciliation. However, StarRocks best practices strictly require BE/CN to be upgraded before FE. * Is 4.1.1 FE → 4.0.8 CN safe? No, it is generally considered unsafe. While StarRocks maintains strong backward compatibility (new CNs can talk to old FEs), forward compatibility (new FE talking to old CNs) is not guaranteed. A 4.1.1 FE might issue RPCs or metadata instructions that a 4.0.8 CN does not understand, leading to crashes or the "wedged" state you previously experienced. * How to force CN-first in K8s: To honor the safe upgrade path using the operator, do not update the

image

for both components in the

StarRocksCluster

CR at once. 1. Step 1: Patch only the

starRocksCnSpec.image

to 4.1.1. 2. Step 2: Monitor the CN pods (

kubectl get pods -w

) until all CNs are

Running

and

Ready

on 4.1.1. 3. Step 3: Only then patch the

starRocksFeSpec.image

to 4.1.1. This "manual sequencing" via the CR ensures you follow the validated

CN -> FE

path. Summary of Quiesce Configs (4.1.x) To ensure a clean "straddle-free" window, use these exact commands:

sql
-- Stop tablet scheduling/balancing
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "0");
ADMIN SET FRONTEND CONFIG ("tablet_sched_max_balancing_tablets" = "0");
ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");

-- Stop background data maintenance
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
ADMIN SET FRONTEND CONFIG ("lake_enable_tablet_split" = "false"); -- Use lake_ prefix

-- Stop MV and Stats
ADMIN SET FRONTEND CONFIG ("materialized_view_max_refresh_thread_num" = "0");
ADMIN SET FRONTEND CONFIG ("enable_statistic_collect" = "false");

One final tip: When you are ready to roll the FEs, verify that

SHOW PROC '/transactions';

is entirely empty or only shows

VISIBLE

transactions. If any are stuck in

PREPARED

or

COMMITTED

for more than a few minutes, resolve them before changing the FE version. References • administration/management/FE_configuration.md • deployment/upgrade.md • deployment/sr_operator.md • release_notes/release-4.1.md References • administration/management/FE_configuration.md