< Rocky> I am running into a compaction publish race conditi StarRocks #questions-and-troubleshooting

<@U0A71G31CDV>, I am running into a compaction pub...

kushagra.dubey

07/28/2026, 6:51 AM

@Rocky, I am running into a compaction publish race condition on starrocks, because of which there are lot of transactions stuck in

COMMITED

state which are not getting finished. Starrocks-fe is logging this error - "2026-07-27 201807.121Z ERROR (publish-task-34|424) [PublishVersionDaemon.publishPartitionBatch():601] publish partition batch partition.getVisibleVersion() + 1 != version.get(0) 1774493 49 49",

Rocky

07/28/2026, 6:51 AM

This error message

publish partition batch partition.getVisibleVersion() + 1 != version.get(0)

indicates a version discontinuity (gap) or a stale transaction issue during the publish phase in a StarRocks shared-data cluster. In your case, the log shows that the system expects the next version to be

, but it encountered a transaction (likely an old compaction) with version

. This discrepancy blocks the

PublishVersionDaemon

from finishing the transaction, leaving it and all subsequent transactions for that partition stuck in the

COMMITTED

state. Root Cause This is a known race condition in shared-data mode (often triggered during FE leader changes or concurrent Schema Change/Compaction jobs). A transaction is committed with a version that does not align with the partition's

VisibleVersion + 1

. Because StarRocks requires strict version continuity for publishing, the "gap" or "outdated" version stops the publish pipeline. Recommended Troubleshooting & Fixes 1. Identify the Stuck Transactions and Partitions Find the specific table and partition causing the issue. * Check transactions: Run

SHOW PROC '/transactions'

to find transactions stuck in the

COMMITTED

state. Note their

TxnId

and the affected

PartitionId

. * Check partition versions: Run

SHOW PARTITIONS FROM <table_name>

and compare

VisibleVersion

NextVersion

, and

CompactVersion

. 2. Clear Stuck Compaction Tasks If the "version 49" in your log belongs to an old compaction task: * Run

SHOW PROC '/compactions'

to see if there are any tasks for that partition with an empty

FinishTime

. * Cancel the problematic compaction transaction:

Copy code

sql
    CANCEL COMPACTION WHERE TXN_ID = <txn_id>;

3. Disable Batch Publishing (Temporary Workaround) Some versions of StarRocks have a bug in the batch publishing logic. Disabling it can sometimes allow the publisher to recover or handle transactions individually:

Copy code

sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

Note: This may impact ingestion throughput, so monitor performance. 4. Manually Align Partition Version (High Risk) If the metadata is inconsistent (e.g., the partition version is "stuck" at a lower number than the data actually present), you can force the version forward. Warning: Use this only if you are certain the data for the higher version exists in storage.

Copy code

sql
ADMIN SET TABLE <table_name> PARTITION (<partition_name>) VERSION TO <expected_version>;

5. Clean up stale BE/CN tasks In shared-data mode, FE-scheduled tasks can sometimes leave stale state on Compute Nodes. 1. Set

lake_compaction_max_tasks

to stop new compactions. 2. Restart the BE/CN nodes to clear memory-resident stuck tasks. 3. Set

lake_compaction_max_tasks

back to

-1

(default). Permanent Fix This issue has been addressed in newer maintenance releases. If you are on an older version of StarRocks (e.g., 3.1.x or early 3.2.x), consider upgrading to at least v3.3.x or v3.4.x, where the version management and lock contention logic for shared-data compaction have been significantly improved. If the issue persists after these steps, please provide the

PartitionId

from the log and the output of

SHOW PARTITIONS

for the affected table in the StarRocks community Slack for a for further investigation. References • faq/shared_data_faq.md • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md

kushagra.dubey

07/28/2026, 6:57 AM

@Rocky, Current starrocks version - 4.0.8; And

SHOW PROC '/compactions';

returns around

rows of different tables and all of them are in COMMITED state

Rocky

07/28/2026, 7:01 AM

The error message

publish partition batch partition.getVisibleVersion() + 1 != version.get(0) 1774493 49 49

indicates a version sequence violation in your shared-data cluster. Specifically: * Expected Version:

(This is the partition's

VisibleVersion + 1

). * Actual Version Received:

. * The Problem: The

PublishVersionDaemon

is trying to publish a transaction that was assigned version

years/eons ago, but the partition is already at version

. In StarRocks, versions must be published in strict incremental order. This "ghost" transaction from version 49 is blocking the entire publish queue, which is why your 1,000+ other transactions are stuck in the

COMMITTED

state. Step 1: Identify and Remove the Stale Transaction You need to find the specific

TxnId

associated with that version

for the affected partition. 1. Check the transactions list:

Copy code

sql
    SHOW PROC '/transactions';

Look for a transaction with a very old

CreateTime

or one that corresponds to the partition ID mentioned in your FE logs (the log likely mentions a

partitionId

before the error message). 2. Identify the compaction task: Since you see 1,047 rows in

SHOW PROC '/compactions'

, find the one with the smallest

TxnID

or one that has a

CommitTime

but no

FinishTime

and appears to be the "oldest." 3. Cancel the stale transaction: If you find a

TxnID

for that old version 49:

Copy code

sql
    CANCEL COMPACTION WHERE TXN_ID = <stale_txn_id>;

Or, if it's a regular load:

Copy code

sql
    ROLLBACK TRANSACTION <stale_txn_id>;

Step 2: Clear the "COMMITTED" Backlog Once the blocking stale transaction is removed, the publisher should ideally resume. However, with 1,000+ stuck tasks, the system might struggle to catch up. 1. Temporarily disable the batch publisher: There is a known issue where the batch publish logic gets stuck on these gaps. Switching to single-transaction publishing can sometimes bypass the validation error and allow the system to skip the invalid version.

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

Wait a few minutes to see if the

COMMITTED

count decreases. 2. Force-clear compaction tasks: If the

SHOW PROC '/compactions'

list does not clear: * Disable new tasks:

Copy code

sql
        ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");

* Restart all BE/CN nodes: This clears the memory-resident task state on the compute nodes. * Re-enable:

Copy code

sql
        ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");

Step 3: Repair Partition Metadata (High Risk) If the FE still thinks it needs to publish version 49 but version 49 is long gone or invalid, you may need to force the partition metadata to align. Use this only if the table is currently unusable. Find the

partition_id

from your logs and run:

Copy code

sql
-- Replace <version> with the expected next version (e.g., 1774493)
ADMIN SET TABLE <table_name> PARTITION (<partition_id>) VERSION TO <version>;

Note: This command aligns the metadata to let the system skip the gap, but ensure you are targeting the correct partition ID. Summary Recommendation for v4.0.x Since you are on a very high version (v4.0.8), this might be a regression or a side-effect of a failed Schema Change. If

lake_enable_batch_publish_version = false

doesn't clear the queue within 10 minutes, the most effective way to recover the metadata is to use ADMIN REPAIR TABLE
:

Copy code

sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>)
PROPERTIES ("enforce_consistent_version" = "true");

This will force the FE to re-sync the tablet versions from the shared storage and should resolve the "VisibleVersion + 1" mismatch. References

Rocky

07/28/2026, 7:01 AM

• sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_SET_PARTITION_VERSION.md • sql-reference/sql-statements/cluster-management/tablet_replica/ADMIN_REPAIR.md • administration/management/resource_management/Replica.md • sql-reference/sql-statements/table_bucket_part_index/ALTER_TABLE.md • table_design/data_distribution/dynamic_partitioning.md

kushagra.dubey

07/28/2026, 8:22 AM

@Rocky, I ran the REPAIR command on one of the table after which compaction tasks reduced and running transactions were also dropping but now I can see lot of BACKEND_STREAMING transactions which are in stuck state and the ones that got finished are in aborted state.

Rocky

07/28/2026, 8:22 AM

The transition of transactions to

ABORTED

or stuck

BACKEND_STREAMING

states after a metadata repair often happens because the ingestion clients (like Flink or Routine Load) are still attempting to write data based on the old version sequence that was present before the repair. In StarRocks shared-data clusters, transactions must be published in a strict, continuous sequence. If

REPAIR

forced the partition version forward to bypass the "version 49" gap, any streaming transaction that started before the repair now has an invalid "base version" and will be rejected by the FE. 1. Identify the Abort Reason Run the following to see why the

BACKEND_STREAMING

tasks are failing:

Copy code

sql
-- Replace <db_name> with your database name
SHOW PROC '/transactions/<db_name>';

Look at the Reason
and ErrorMsg
columns. You will likely see: * `VERSION_ALREADY_EXIST`: The load is trying to commit a version the system has already surpassed. * `PUBLISH_VERSION_TIMEOUT`: The publishing queue is still too long. * `PREVIOUS_TRANSACTION_NOT_FINISHED`: There is still one older transaction (likely a

COMMITTED

one) blocking that specific partition. 2. Immediate Recovery Steps Step A: Restart Ingestion Clients Streaming transactions (especially from Flink or Routine Load) cache metadata. If you repaired the table metadata, the existing stream handles are likely stale. 1. Pause your Routine Loads or Stop your Flink jobs. 2. Wait 1 minute for all active

BACKEND_STREAMING

transactions to clear or abort. 3. Resume/Restart the jobs. This forces them to fetch the new

VisibleVersion

from the FE. Step B: Clear any remaining "COMMITTED" Streaming Txns If there are still transactions stuck in

COMMITTED

that aren't moving: 1. Find the

TxnId

from

SHOW PROC '/transactions'

. 2. Abort them manually (they cannot be saved if the version sequence has moved past them):

Copy code

sql
    -- For regular loads
    ROLLBACK TRANSACTION <txn_id>;

Step C: Tune the Publisher for the Backlog With 1,000+ tasks recently cleared, the FE's internal

PublishVersionDaemon

might be struggling. Apply these temporary settings to help it stabilize:

Copy code

sql
-- Disable batching to process transactions one by one (more stable during recovery)
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Increase the timeout to prevent "Aborted" status during heavy backlog processing
ADMIN SET FRONTEND CONFIG ("publish_version_timeout_ms" = "60000");

3. Verify Final Metadata Health Check the partitions of the affected tables:

Copy code

sql
SHOW PARTITIONS FROM <table_name>;

Ensure that for every partition: *

VisibleVersion + 1 == NextVersion

. * If

NextVersion

is much higher than

VisibleVersion

, the system is still publishing. Wait for them to align. Why did this happen? On StarRocks v4.0.8, the "version 49" error you saw usually stems from a Schema Change race condition or a failed Compaction commit that left a "hole" in the version sequence. When you ran

REPAIR

, you effectively "jumped" over the hole, but the streaming jobs were still trying to fill it, or were confused by, the old versioning logic. If the

ABORTED

error message specifically mentions "tablet is in error state" or "invalid rssid", it means the physical data files in object storage are inconsistent with the new metadata, and you may need to

TRUNCATE

and reload the specific affected partition. References • faq/loading/Loading_faq.md • knowledge/admin_and_operation/shared_data_compaction.md

kushagra.dubey

07/28/2026, 8:27 AM

If after the repair command there is version missmatch for lot of tables, for 1 of the the table - compaction version is 59, visible version is 104 and next Version is 122, and there are many tables like this @Rocky

Rocky

07/28/2026, 8:27 AM

The situation you are describing—where

VisibleVersion

(104) is significantly behind

NextVersion

(122)—confirms a Version Discontinuity (Gap). In StarRocks shared-data mode, the

PublishVersionDaemon

is strictly sequential. It is currently looking for transaction version 105 to publish. However, version 105 is either missing, failed to commit properly, or is held up by a specific job (like a Schema Change). Because 105 is not "ready," versions 106 through 122 are stuck in the

COMMITTED

state. This is a known race condition (Issue #64757) usually triggered by concurrent Schema Changes and heavy writes. Step 1: Emergency Cleanup (Stop the Bleeding) If you keep your Flink/Routine Load running, they will continue to create new transactions, pushing the

NextVersion

higher and filling the FE's transaction memory. 1. Stop all ingestion (Flink jobs, Routine Loads, Stream Loads). 2. Clear the transaction backlog:

Copy code

sql
    -- This helps the publisher try to process transactions one-by-one
    -- instead of getting stuck on a large batch containing the gap.
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

Step 2: Identify the Blocking Job The "missing" version (in your case, 105) is often reserved by a background job. 1. Check for Schema Changes:

Copy code

sql
    SHOW ALTER TABLE COLUMN;

Look for any job in the

RUNNING

FINISHED_REWRITING

, or

WAIT_PUBLISH

state. If a schema change job was assigned version 105 and it's stuck, it will block everything. 2. Check for stuck Compactions:

Copy code

sql
    SHOW PROC '/compactions';

If a compaction task was assigned the missing version, you must cancel it:

Copy code

sql
    CANCEL COMPACTION WHERE TXN_ID = <txn_id_of_missing_version>;

Step 3: Force the Version Forward (The Fix) If there are no active jobs and the gap persists, you must manually "jump" the

VisibleVersion

to match the next available version in the queue. Warning: This involves metadata manipulation. Ensure you have stopped writes before doing this. 1. Find the current gap: For the table where

VisibleVersion

is 104 and

NextVersion

is 122:

Copy code

sql
    -- Replace <version> with one less than the NextVersion (e.g., 121)
    -- This tells the FE that the partition is now at 121,
    -- and the next thing it should publish is 122.
    ADMIN SET TABLE <table_name> PARTITION (<partition_name>) VERSION TO 121;

Note: If the table has many partitions, you may need to check
SHOW PARTITIONS
to see which specific one has the gap. 2. Verify: Run

SHOW PARTITIONS

again. The

VisibleVersion

should now be 121, and the publisher should immediately pick up the transaction for version 122. Step 4: Recover the BACKEND_STREAMING Tasks The transactions stuck in

BACKEND_STREAMING

are likely from your Flink/Routine Load jobs that were interrupted. 1. Once you have jumped the version gap, many of these will likely move to

ABORTED

because their "base version" is now invalid. This is expected. 2. Restart your ingestion jobs. They will negotiate a new base version with the FE and start fresh. Prevention for v4.0.8 To prevent this race condition from recurring while you are on this version: 1. Avoid Schema Changes during peak ingestion. 2. Reduce compaction pressure: If you have 1,000+ compaction tasks, your cluster is likely struggling with small files.

Copy code

sql
    -- Temporarily limit concurrent compactions to reduce lock contention
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "10");

3. Upgrade: This specific sequential publish bug is a high priority. Check for a v4.0.x patch release or consider moving to a stable v3.3.x / v3.4.x maintenance release if the v4.x branch remains unstable for your workload.

Rocky

07/28/2026, 8:27 AM

If the
ADMIN SET ... VERSION TO
command fails or doesn't clear the
COMMITTED
status, please grab the
TxnId
of the oldest committed transaction from
SHOW PROC '/transactions'
and contact StarRocks community Slack for a maintainer for a metadata check. References • knowledge/trouble_shooting/fe_deadlock_problems.md

kushagra.dubey

07/28/2026, 8:55 AM

@Rocky, I have dropped two talbes which were among the tables with version missmatch, but I still see compactions jobs on those tables in

SHOW PROC '/compactions';

Rocky

07/28/2026, 8:55 AM

It is common in StarRocks (especially in shared-data/lake mode) for compaction tasks to linger in the

SHOW PROC '/compactions';

output even after a table has been dropped. This happens because the Frontend (FE) maintains an in-memory state of active and recently completed transactions. Here is how to handle these orphaned compaction tasks: 1. Check the Task State In the output of

SHOW PROC '/compactions';

, look at the CommitTime
and FinishTime
columns: * If
FinishTime
is NOT NULL: You are likely just seeing the history of completed tasks. By default, StarRocks keeps a history of the last few compaction tasks. _ _Check config:*

lake_compaction_history_size

(default 12) and

lake_compaction_fail_history_size

(default 12) control how many entries are kept. These will eventually cycle out. * If
FinishTime
is NULL but
CommitTime
is NOT NULL: These are "Committed" but "Unpublished" transactions. Since the table is dropped, the

PublishVersionDaemon

cannot find the table metadata to finish the version update, so they are stuck. 2. Manually Cancel the Orphaned Tasks If the tasks are not finishing and are clogging your queue, you should manually cancel them using their transaction IDs:

Copy code

sql
-- Find the TxnID from SHOW PROC '/compactions'
CANCEL COMPACTION WHERE TXN_ID = <txn_id>;

*Note: This must be executed on the Leader FE node.* 3. Clear the FE Memory (If Stuck) If

CANCEL COMPACTION

does not remove them (sometimes it fails if the table metadata is already purged from the recycle bin), you can force a refresh of the FE's compaction scheduler state: 1. Stop the scheduler temporarily:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");

2. Wait a few minutes to see if the orphaned entries disappear. 3. Resume the scheduler:

Copy code

sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1"); -- -1 restores default auto-calculation

4. Why this matters for your "Version Mismatch" issue Even though you dropped the tables, these stuck

COMMITTED

transactions can still occupy slots in the FE's transaction manager. If you have a total of

rows in your compaction list and most are

COMMITTED

but not

FINISHED

, you may hit the internal limit for concurrent transactions, which will block new tables from publishing data. Recommendation: If the number of rows in

SHOW PROC '/compactions'

does not decrease after manual cancellation and the config change above, restart the Leader FE. This will clear the in-memory transaction state for the deleted tables and allow the publisher to focus on your remaining healthy tables. Summary Checklist 1. Cancel by ID:

CANCEL COMPACTION WHERE TXN_ID = ...

2. Lower History: If you just want a cleaner view:

ADMIN SET FRONTEND CONFIG ("lake_compaction_history_size" = "2");

3. Restart Leader FE: If "COMMITTED" tasks for dropped tables persist and block other work on other tables. References • knowledge/admin_and_operation/shared_data_compaction.md • administration/management/compaction.md

7 Views

Open in Slack

Previous Next