@Rocky I started a process of decommissioning a BE node in my shared nothing cluster. There was spike in transaction publish latency from ~4s to 1 minute. However, update rowset commit apply duration is steady. Compaction score also relatively low. However, the running transactions went from ~10 to spiking between 10 and 40 now like a zig zag pattern. There are 0 unhealthy tablets but in cluster_balance/pending_tablets, I have a few tablets that have inter node tablet distribution status in pending state with error message consecutive full cone needs to delay. The node is fully decommissioned. Why do I still have elevated transactions?

The behavior you're seeing is a known side effect of tablet rebalancing/migration that can occur during or after decommissioning. Even though the node is "fully decommissioned," the system is likely still struggling to finalize the tablet distribution across the remaining nodes. 1. Why you see "consecutive full clone needs to delay" This message indicates that the FE Tablet Scheduler has attempted to clone these specific tablets multiple times, but the tasks failed. To prevent a "retry storm" that could overwhelm BEs, StarRocks applies a backoff delay (cooldown) before trying again. * Cause: This usually happens if the source replica (the node providing data) or the destination replica is under high load, has disk issues, or if there is a version mismatch that makes a incremental clone impossible, forcing a "full clone" which then fails due to timeout or resource limits. * Relation to Decommissioning: When you decommissioned the node, StarRocks had to move all its tablets. If the remaining nodes were already busy or if specific tablets were large/complex, some clones might have failed, leading to these "pending" tablets. 2. Why Transaction Publish Latency Spiked In StarRocks, the Publish Version phase occurs after data is written and committed. For a transaction to be marked as

VISIBLE

, the FE must ensure that a quorum of replicas for every affected tablet has successfully applied the new version. * The Blockage: If a tablet is currently being cloned (or is stuck in "pending"), its replicas may be in an "incomplete" or "catching up" state. The Publish Version daemon may wait for these tablets to reach a consistent state or encounter timeouts while trying to communicate with replicas that are in flux. * The Zig-Zag Pattern: Your running transactions are likely hitting the

tx_visible_wait_timeout

(default 10s) or the FE's internal publish timeout. They pile up while waiting for the stuck tablets to "publish," then they either time out (failing or remaining in a pending state) or finally succeed in a batch, creating the zig-zag pattern. 3. Why it persists after decommissioning is "done" A BE node is marked as decommissioned once its tablets have been reassigned and the node is removed from the cluster metadata. However, the physical migration (the background cloning) and the cleanup of the cluster balance can continue. If those "pending tablets" in

cluster_balance

belong to tables you are currently loading into, they will continue to delay every transaction that touches them. Recommended Troubleshooting Steps 1. Identify the specific failure: Check the FE leader's

fe.log

for the specific tablet IDs mentioned in

pending_tablets

. Look for messages containing

clone

and the tablet ID to see the actual error (e.g.,

timeout

,

no source found

,

disk full

). 2. Inspect the stuck tablets: Run the following to see the status of the problematic tablets:

sql
    SHOW TABLET <tablet_id>;

Check the

Version

and

IsBad

columns. If the version is significantly behind the

VisibleVersion

of the table, that tablet is blocking your loads. 3. Tuning to clear the backlog: If your BEs have available overhead, you can temporarily increase the scheduling concurrency to clear the pending clones faster:

sql
    -- Increase slots per disk for clone tasks (default is usually 2 or 8)
    ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "16");
    -- Increase max concurrent scheduling
    ADMIN SET FRONTEND CONFIG ("tablet_sched_max_scheduling_tablets" = "2000");

4. Reduce Publish Wait (Emergency Mitigation): To stabilize the transaction latency immediately, you can tell the FE not to wait as long for non-essential replica synchronization during publish: ```sql -- Setting this to 0 makes the publish return as soon as quorum is met -- without waiting for all replicas to acknowledge.

@Rocky Can you explain in more detail? For reference, I do have a table with no replication. But perhaps that is not causing the issue?

The fact that you have a table with no replication (

replication_num = 1

) is almost certainly the root cause of your issue. In a shared-nothing cluster, single-replica tables create a "zero-fault-tolerance" environment that behaves poorly during decommissioning. 1. The "Decommissioning" Paradox When you decommission a node, StarRocks attempts to move all its data to other nodes before the node is officially removed. * For 3-replica tables: This is safe. The FE clones the data from the other two healthy replicas to new nodes. * For 1-replica tables: There is no second source. To move the data, the FE must perform a full clone from the decommissioned node to a new node. If you see the node is "fully decommissioned" (i.e., it no longer appears in

SHOW BACKENDS

), but you still have tablets in

pending_tablets

with cloning errors, the system is in a "lost data" or "stuck" state. The metadata thinks the node is gone, but the data migration for those single-replica tablets never finished. 2. Why "Consecutive full clone needs to delay"? This error means the FE tried to move those specific single-replica tablets, but the task failed repeatedly. * Why it fails: If the source node (the one being decommissioned) became unresponsive, was shut down too early, or hit a timeout during the 1-minute transaction spikes, the clone fails. * The Delay: StarRocks won't keep hammering a failing clone. It puts the tablet into a "cooldown" (the delay you see). * The Trap: Because these tablets have only 1 replica, if that replica was on the decommissioned node and hasn't successfully moved yet, the tablet is effectively offline. 3. Why Transactions are Spiking (The Zig-Zag) In StarRocks, the Publish Version phase must succeed for every tablet involved in a transaction. * The Wait: Even if a table has only 1 replica, the FE must wait for that single replica to acknowledge it has applied the data. * The Block: If that tablet is currently "Pending" or "In Delay" due to the failed migration, the FE doesn't know where to send the version update, or it sends it to a replica that isn't ready. * The Zig-Zag: Your transactions are likely hitting the

tablet_publish_timeout

(default usually 30s-60s). They queue up (running transactions go up), wait for the timeout, and then fail or partially succeed, causing the drop in the zig-zag. 4. Why "0 Unhealthy Tablets"? The

unhealthy

count in some views specifically tracks if a tablet has fewer than a quorum of replicas. * For a 1-replica table, the "quorum" is 1. * If the FE still thinks the replica exists (even if it's on a decommissioned node or in a ghost state), it might not flag it as "unhealthy" immediately, but it will show up as "pending" in the

cluster_balance

or

tablet_scheduler

because it's not where it's supposed to be. Recommended Actions 1. Check if data is lost: Run

SHOW TABLET <id>

for one of the pending tablets. Look at the

BackendId

. Is that Backend actually alive in

SHOW BACKENDS

? If that BE is gone, you may have a "dead" tablet. 2. Increase Replication (Prevention): You should generally never use

replication_num = 1

for production data. If you can, increase it to 3:

sql
    ALTER TABLE <table_name> SET ("replication_num" = "3");

Note: This will only work if you still have the data source available. 3. Identify the Table: Run this to find which tables are stuck:

sql
    SELECT TABLE_NAME, TABLET_ID FROM information_schema.be_tablets WHERE STATE = 'DECOMMISSION';

4. Force Repair (If data is still on disk): If the decommissioned node is physically still there but "dropped" from StarRocks, you might need to re-add it briefly to allow the FE to finally finish the clones for those 1-replica tablets.

@Rocky Can you further explain why 3 replica tables tablet rebalance must faster than no replica tables? Is it because of competition between writes and clones?

The "zig-zag" pattern and the spike in transaction latency are directly related to how StarRocks handles data consistency vs. data movement when there is no redundancy. Rebalancing a 3-replica table is significantly faster and more stable than a 1-replica table for three primary reasons: 1. Quorum "Blocking" vs. Parallelism * 3-Replica Tables: When a transaction finishes, it follows the

write_quorum

(usually

MAJORITY

). If one replica is busy being moved (cloning), the transaction only needs acknowledgement from the other two replicas to succeed. The third replica can catch up in the background without slowing down your transactions. * 1-Replica Tables: The

write_quorum

is effectively ALL (1 out of 1). For a transaction to commit, that single replica must be available. If that replica is currently occupied by a "Full Clone" task (moving data to a new node), it creates a resource lock. The FE's Publish Version daemon must wait for the cloning task to yield or finish before it can update the version, leading to the 1-minute latencies you observed. 2. Full Clone vs. Incremental Clone * The 1-Replica Bottleneck: Because there is only one copy of the data, the system must perform a "Full Clone" from the decommissioned node. A full clone is a heavy I/O operation that copies the entire physical data directory. * The 3-Replica Advantage: If you decommission a node in a 3-replica setup, the system often has multiple sources to pull from. More importantly, it can use "Incremental Clones" to sync small deltas if a replica is just slightly behind, which is much faster than a full disk-to-disk copy. 3. Resource Competition (The "Zig-Zag" Cause) The zig-zag pattern in your "running transactions" is a symptom of a Timeout & Retry loop: 1. Saturation: The BE node is at 100% disk/network I/O because it's trying to push out a Full Clone of your 1-replica tablets. 2. Pile-up: New transactions arrive and try to "Publish Version." Since the BE is saturated, these tasks sit in a queue. Your "running transactions" count spikes from 10 to 40. 3. Timeout: The transactions hit the

tablet_publish_timeout

(or

tx_visible_wait_timeout

). They fail or the FE gives up on that tablet, and the "running" count drops sharply. 4. Backoff (The Delay): Because the clone failed (due to the timeout/congestion), the Tablet Scheduler triggers the "consecutive full clone needs to delay" safety mechanism. This pauses the move for a few minutes, which briefly lowers latency until the next clone attempt starts the cycle over again. Why the node is "Decommissioned" but the issue persists In StarRocks, a node is marked "Decommissioned" once the FE metadata has reassigned all its tablets to other nodes. However, the *physical cleanup*—deleting the old data and finishing the last "pending" version updates on the new destination nodes—can still be in progress. If you still have tablets in

pending_tablets

with error messages, the cluster is not yet stable. How to clear the backlog: If the latency is unacceptable, you can force the FE to stop waiting for these "ghost" replicas during the publish phase:

sql
-- This tells the FE to return success as soon as the quorum is met. 
-- For 1-replica, this won't help much, but it prevents the FE from 
-- waiting on replicas that are in an 'incomplete' state.
ADMIN SET FRONTEND CONFIG ("quorum_publish_wait_time_ms" = "0");

Long-term fix: You should avoid

replication_num = 1

for any table that receives constant writes. Without at least 2 replicas, every maintenance task (like decommissioning) becomes a "stop-the-world" event for that table's writes. References • administration/management/resource_management/Replica.md • administration/management/BE_configuration.md

@Rocky So basically there is competition between writes and cloning in 1-replica tables. So there is resource lock

Exactly. Your intuition is spot on. In a 1-replica scenario, you have a hard resource and logical contention that doesn't exist with 3 replicas. Here is the technical breakdown of that "lock" and competition: 1. The Write Quorum "Dead End" In StarRocks, a transaction is successful only when the

write_quorum

is met. * 3-Replica Table: The quorum is typically

MAJORITY

(2 out of 3). If one replica is busy being cloned or moved, the transaction can still commit to the other two. The third replica catches up later. * 1-Replica Table: The quorum is 1 out of 1. If that single replica is locked or occupied by a system task, the entire transaction must wait. There is no "other" replica to take the load. 2. Physical vs. Logical Competition When you decommission a node with a 1-replica table, the following competition occurs: * The Full Clone (System Task): Since there is only one copy, the BE must perform a "Full Clone." This is a heavy, physical I/O operation that copies the entire tablet data folder. During certain phases of this clone (especially the final metadata "commit" of the clone), the tablet is often marked as

CLONE

or

DECOMMISSION

state, which can restrict its availability for writes. * The Publish Version (User Task): Every time you load data, the FE sends a

PublishVersion

task to the BE. The BE must apply the new data and update the tablet's version metadata. * The Conflict: If a Full Clone is saturating the disk I/O or holding a internal lock on the tablet's metadata to finalize the migration, the

PublishVersion

task gets queued. 3. Why the Latency Spikes (The 1-Minute Wait) The "zig-zag" and the 1-minute spikes are caused by timeout thresholds: 1. Queueing: Your transactions pile up while the BE is busy cloning the 1-replica tablet. 2. Wait: The FE waits for the BE to acknowledge the

PublishVersion

. By default, the FE will wait up to the

tablet_publish_timeout

(often 30s-60s). 3. Failure & Retry: If the clone takes too long, the Publish task times out. The FE then retries or marks the transaction as failed, causing the "running transactions" count to drop suddenly (the zig-zag). 4. "Consecutive Full Clone Needs to Delay" This message is the FE's safety valve. It means: * The FE tried to move your 1-replica tablet. * The task failed (likely because the BE was too busy handling your "zig-zagging" write transactions). * Instead of retrying immediately and making the congestion worse, the FE puts that tablet into a cooldown period (the "delay"). During this delay, the tablet remains on the "decommissioned" node or in a limbo state, and your transaction latency stabilizes briefly until the FE tries to clone it again. Summary of the "Lock" With 1 replica, cloning is a blocking operation for writes. You are essentially trying to change the tires on a car while it's driving at 60mph. With 3 replicas, you have "spare tires" (the other replicas) that allow the car to keep moving while one tire is being swapped. Immediate Mitigation: If you must finish this decommission while keeping the cluster alive: 1. Pause or throttle your data ingestion for a few minutes. This removes the competition, allowing the "Full Clone" to finish without being interrupted by

PublishVersion

requests. 2. Increase clone slots to prioritize the move:

ADMIN SET FRONTEND CONFIG ("tablet_sched_slot_num_per_path" = "8");

3. For the future: Always use

replication_num = 3

for production. It is the only way to ensure maintenance tasks like decommissioning don't impact transaction latency-lock your transactions. References • knowledge/usage/load/shared_data_batch_ingestion.md • deployment/shared_data/feature-support-shared-data.md