@Kevin Cai @Rocky i have this issue please take a look: The problem in one sentence Two load transactions on

TB_INVOICE

are stuck forever in COMMITTED because the partition version moved far ahead of them, and the tablet scheduler refuses to drop extra replicas until all older transactions on that table are finished. The two zombie transactions TxnIdStatusCommit versionPartition visibleError21722630 COMMITTED 640 1166

wait for publishing partition 13427821 version 1167. self version: 640

21736671 COMMITTED 641 1166

wait for publishing partition 13427821 version 1167. self version: 641

They are zombies because: • FE expects

visibleVersion == commitVersion - 1

before publish (needs visible 639 or 640) • Actual visible is 1166 — about 525 versions ahead •

finishTransaction()

returns and retries forever; they never become

VISIBLE

or

ABORTED

SHOW PROC '/transactions/16625/running'; -- Only these 2 rows — both COMMITTED, no FinishTime Why 1,618 balance tasks are pending When StarRocks drops a REDUNDANT replica (3rd, 4th, … copy), it does this: 1. Mark replica DECOMMISSION, set watermark txn id (e.g. 22146433) 2. Wait: all running txns on THAT TABLE with txn_id ≤ watermark must finish 3. Then delete the replica Step 2 calls

isPreviousTransactionsFinished()

. COMMITTED counts as “running”. Your zombies have low IDs (21722630, 21736671) on table 13422479. Every REDUNDANT drop on

TB_INVOICE

hits: wait txn before 22146433 to be finished The watermark is not the bug — the unfinishable COMMITTED txns are. The scheduler retries thousands of times (

FailedSched

711, 2168, 2520…) and stays PENDING. Why normal commands cannot fix it (3.5.16 shared-nothing) MethodResultWait / timeout COMMITTED txns do not time out

abortTransaction

Rejected — already COMMITTED

ADMIN SKIP COMMITTED TRANSACTION

Not in 3.5.16 (lake-only in newer versions) Restart FE/BE State reloads from journal — still COMMITTED Decommission be-8 Does not clear these txns

tablet_sched_always_force_decommission_replica=true

Does not bypass wait for replicas already in DECOMMISSION Data safety • Partition is at version 1166; these txns wanted 640/641 • Their data was never published and is already bypassed • Clearing them is metadata cleanup — not rolling back current table data How to resolve (ranked) 1. StarRocks support (safest operationally) Send them txn IDs 21722630, 21736671, table 13422479, partition 13427821, version 3.5.16 shared-nothing, and 1,618 pending tasks. Ask for an official way to finalize or abort orphan COMMITTED txns on OLAP tables. 2. Custom FE patch (best technical fix) In

DatabaseTransactionMgr.finishTransaction()

(~line 1174 on branch-3.5): when

visibleVersion >= commitVersion

, treat the partition as a no-op (same path as dropped partition), let the txn reach VISIBLE, remove it from the running map. Then rolling restart FEs → balance queue should start draining within hours. 3. BDB-JE journal injection (no rebuild, higher risk) Stop leader FE, backup

meta/bdb/

, inject journal entry with full

TransactionState

JSON setting

transactionStatus=ABORTED

+

finishTime

. Only with support guidance or after full backup.

This is a highly specific "version gap" scenario in StarRocks 3.5.16 (shared-nothing) where the transaction publish logic is stuck in a retry loop. Your diagnosis is correct: the FE’s

finishTransaction

logic expects sequential version publishing (

visibleVersion == commitVersion - 1

). When the visible version jumps ahead (e.g., due to other concurrent loads or manual intervention), these older

COMMITTED

transactions become "orphans" that block the tablet scheduler’s watermark checks. In version 3.5.16, there is no built-in SQL command (like

ADMIN SKIP COMMITTED TRANSACTION

) to clear these specifically for shared-nothing storage. Recommended Resolution Path 1. Official Support / Developer Assistance (Highly Recommended) Since this involves direct metadata manipulation or a binary patch, you should provide the specific transaction IDs and table/partition info to the StarRocks community or support. They can provide a tailored metadata cleanup script or a hotfix. 2. Technical Fix: FE Source Patch Your proposed fix for

DatabaseTransactionMgr.finishTransaction()

is the most direct technical solution. The logic needs to handle the case where the partition version has already surpassed the transaction's target version. * Target:

DatabaseTransactionMgr.java

(near the

finishTransaction

or

publishVersion

logic). * Logic: If

partition.getVisibleVersion() >= transactionState.getCommitVersion()

, the transaction should be treated as successfully published (metadata-only) because its data has been effectively superseded or "lost" to the future version. This allows the transaction to move to the

VISIBLE

state and be removed from the running transaction map. * Execution: 1. Apply the patch to a custom FE build. 2. Rolling restart FEs. 3. Once the FEs are up, they should eventually process these transactions, moving them to

VISIBLE

. 4. The tablet scheduler's

isPreviousTransactionsFinished()

check will then pass, allowing the balance queue to drain. 3. BDB-JE Journal Injection (Advanced/High Risk) This is essentially "manual surgery" on the metadata. This is generally discouraged unless performed under guidance from the StarRocks core team. * Risk: If the JSON structure or version in the journal is incorrect, the FE may fail to start or corrupt the metadata image. * Procedure: It requires stopping the FE, using a utility to modify the BDB-JE records to set the

TransactionStatus

to

ABORTED

or

VISIBLE

, and restarting. Always take a full metadata backup (

meta/

) before attempting this. Why other methods fail *

abortTransaction

: Only works before a transaction is

COMMITTED

. Once in

COMMITTED

state, the data is technically "written" to BEs, and FE is committed to publishing it. * Restarting FE: The state is persistent in the edit log. The FE will reload the

COMMITTED

status and immediately resume the failing publish retry loop. *

tablet_sched_always_force_decommission_replica

: This flag helps bypass certain health checks but does not override the safety watermark that ensures data consistency by waiting for all pre-existing transactions to finish before deleting a replica. Data Safety Confirmation As you noted, this is metadata cleanup. Since the partition is already at version 1166, data associated with versions 640 and 641 is physically unreachable and superseded. Clearing these transactions will not affect the current data integrity of

TB_INVOICE

. INVOICE`.

.

VOICE

. References • loading/SQL_transaction.md • release_notes/release-3.5.md • administration/management/compaction.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md

i want that code to starrocks code and have my customized version of it. how can i do it

To build a customized version of StarRocks 3.5.16 to resolve this "zombie transaction" issue, follow these steps. The safest and most consistent way is to use the StarRocks Docker development environment. 1. Clone and Prepare the Source Code You need to target the

branch-3.5

branch which corresponds to your current deployment.

bash
# Clone the repository
git clone <https://github.com/StarRocks/starrocks.git>
cd starrocks

# Checkout the branch for 3.5.x
git checkout branch-3.5

2. Modify the Code Open the following file in your editor:

fe/fe-core/src/main/java/com/starrocks/transaction/DatabaseTransactionMgr.java

Find the

finishTransaction

or

publishVersion

related methods (around line 1170 in 3.5). You want to add logic that checks if the partition's current visible version is already greater than or equal to the transaction's commit version. Pseudo-code logic to add:

java
if (partition.getVisibleVersion() >= transactionState.getCommitVersion()) {
    // Log that this transaction is being skipped/auto-finished because 
    // the partition version has already moved ahead.
    <http://LOG.info|LOG.info>("transaction {} has commit version {} while partition {} has visible version {}. " +
             "Marking as visible to clear orphan txn.", 
             transactionState.getTransactionId(), transactionState.getCommitVersion(), 
             partition.getId(), partition.getVisibleVersion());
    // Continue the logic as if the publish was successful 
    // to move the transaction to VISIBLE state.
}

3. Build using Docker Using the official dev-env image ensures all dependencies (Maven, Thrift, etc.) are correct.

bash
# Pull the 3.5 specific dev environment
docker pull starrocks/dev-env-ubuntu:3.5-latest

# Run the container and mount your source code
docker run -it -v $(pwd):/root/starrocks --name sr-build starrocks/dev-env-ubuntu:3.5-latest /bin/bash

# Inside the container, build ONLY the Frontend (FE) to save time
cd /root/starrocks
./build.sh --fe

4. Deploy the Custom FE Once the build finishes, your customized binary will be located in:

output/fe/lib/starrocks-frontend.jar

To apply the fix: 1. Backup: Back up your existing

fe/lib/starrocks-frontend.jar

on your FE nodes. 2. Stop FE: Stop the leader FE and follower FEs. 3. Replace: Swap the existing

.jar

file with your newly compiled one. 4. Restart: Restart the FEs. Once restarted, the FE should detect the version gap, trigger your new logic, and move those

COMMITTED

transactions to

VISIBLE

. This will clear the "running" transaction map and allow the tablet scheduler to proceed with deleting the redundant replicas. Important Warnings * Version Match: Ensure you are building against the exact tag/branch that matches your BE versions to avoid protocol mismatch. * Metadata Backup: Always perform a full backup of your

meta/

directory before deploying custom code that alters transaction states. * Testing: If possible, test this build on a non-production cluster first to verify the state transition. References • developers/build-starrocks/Build_in_docker.md • developers/build-starrocks/handbook.md • loading/SQL_transaction.md • administration/management/compaction.md action) References • developers/build-starrocks/Build_in_docker.md • developers/build-starrocks/handbook.md • loading/SQL_transaction.md • administration/management/compaction.md