Danila Fediashchin
08/10/2026, 9:49 AM2026-08-10 17:47:39.669+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionBatchAsync():942] start publish lake batch db:58879 table:133024 txns:604254,608854,608852,608853,608856,608857,608858,608861,608863,608864
2026-08-10 17:47:40.335+08:00 ERROR (publish-task-16|353) [PublishVersionDaemon.publishPartitionBatch():739] Fail to publish partition 409133 of txnIds [604254]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[423646, 423649, 423659, 423661, 423667, 423669]], error msg: input rowset 8792 not found, host: event-warehouse-cn-0.event-warehouse-cn-search.event-warehouse.svc.cluster.local
at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:404) ~[fe-core-4.1.3.jar:?]
at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:452) ~[fe-core-4.1.3.jar:?]
at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:436) ~[fe-core-4.1.3.jar:?]
at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:722) ~[fe-core-4.1.3.jar:?]
at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$16(PublishVersionDaemon.java:964) ~[fe-core-4.1.3.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-08-10 17:47:40.344+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionBatchAsync():942] start publish lake batch db:58879 table:133024 txns:604254,608854,608852,608853,608856,608857,608858,608861,608863,608864
2026-08-10 17:47:41.009+08:00 ERROR (publish-task-53|872) [PublishVersionDaemon.publishPartitionBatch():739] Fail to publish partition 409133 of txnIds [604254]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[423646, 423649, 423659, 423661, 423667, 423669]], error msg: input rowset 8792 not found, host: event-warehouse-cn-2.event-warehouse-cn-search.event-warehouse.svc.cluster.local
at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:404) ~[fe-core-4.1.3.jar:?]
at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:452) ~[fe-core-4.1.3.jar:?]
at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:436) ~[fe-core-4.1.3.jar:?]
at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:722) ~[fe-core-4.1.3.jar:?]
at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$16(PublishVersionDaemon.java:964) ~[fe-core-4.1.3.jar:?]
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
at java.lang.Thread.run(Thread.java:840) ~[?:?]
Restarting doesn’t clear this, ADMIN REPAIR TABLE can’t run because Table state is not NORMAL
CN nodes are restarting/reconnectingRocky
08/10/2026, 9:50 AMinput rowset [id] not found during the PublishVersion stage in a shared-data (lake) cluster usually indicates a metadata inconsistency. The FE is attempting to commit a transaction (often a compaction or a batch load) that references a rowset ID which the CN nodes cannot find in their local cache or in the shared storage metadata.
Since your cluster is unresponsive and the table is not in a NORMAL state, follow these steps to diagnose and recover:
1. Identify the Table State
The reason ADMIN REPAIR TABLE fails is that the table is likely locked by an ongoing or stalled background task (e.g., Schema Change, Rollup, or a stuck Compaction).
* Check Schema Changes: SHOW ALTER TABLE COLUMN;
* Check Rollups: SHOW ALTER TABLE ROLLUP;
* Check Table Status: SHOW PROC '/dbs/[db_id]/[table_id]/partitions'; (Use the IDs from your log: DB 58879, Table 133024).
If any task is in a RUNNING or FAIL state, you may need to wait for it to time out or attempt to cancel it.
2. Diagnose the Stuck Transaction
The log points to transaction 604254. Check its current status:
sql
SHOW TRANSACTION WHERE id = 604254;
* If the state is COMMITTED but not VISIBLE, the PublishVersionDaemon is failing to move it to the final state because of the missing rowset.
* If the cluster is completely stuck, look for other transactions that might be blocking the queue: SHOW PROC '/transactions';
3. Clear Stuck Compaction Tasks (Shared-Data)
In shared-data mode, "rowset not found" often happens if a compaction task references rowsets that were already cleaned up or are inconsistent in the metadata.
1. Check for running compactions:
sql
SELECT * FROM information_schema.be_cloud_native_compactions WHERE STATE = 'RUNNING';
2. Temporarily disable compaction to stop the PublishVersionDaemon from retrying the failing task:
sql
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
3. Restart the Leader FE: This often clears the in-memory queue of the PublishVersionDaemon and allows it to skip or re-evaluate the failed batch.
4. Forced Cleanup (If persistent)
If the table remains in a non-NORMAL state and you cannot run any repairs:
* Verify Tablet Health: Run SHOW TABLET [id] for one of the tablets in the error (e.g., 423646). Look at the DetailCmd in the output to see the version chain.
* Skip the transaction (Caution): If a single transaction is corrupting the version chain, you might need to contact StarRocks support or use internal metadata tools to skip that version, as manually editing the starMgr metadata is high-risk.
Summary of Actionable Commands:
1. SHOW ALTER COLUMN FROM [db_name]; -> Check if a schema change is blocking the table.
2. ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0"); -> Stop the retry loop.
3. Restart Leader FE.
4. Once the cluster is responsive, re-enable compaction: ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");.
If CNs are constantly restarting, check the CN <http://be.INFO|be.INFO> logs for ObjectStorageException or OOM which might be triggered by the publish task failing repeatedly.
References
• faq/Sql_faq.md
• data_source/datalake_faq.md
• using_starrocks/async_mv/troubleshooting_asynchronous_materialized_views.md
• faq/shared_data_faq.md
• administration/management/monitoring/alert.md
• data_source/feature-support-data-lake-analytics.mdDanila Fediashchin
08/10/2026, 9:58 AM4.1.3-8a8e186Rocky
08/10/2026, 9:58 AMNORMAL state, it is likely locked by an internal metadata task (typically a Schema Change, Rollup, or an Optimize job) that is either stalled or waiting for the PublishVersion to succeed.
Follow these steps to break the loop and force the table back to a repairable state:
1. Break the Batch Publish Loop
The PublishVersionDaemon is currently grouping multiple transactions (604254, 608854, etc.) into one batch. If transaction 604254 is corrupt, it fails the entire batch.
Disable batch publishing to isolate the problematic transaction and allow others to proceed:
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
Wait a few minutes to see if the other transactions in the log (608854, etc.) become visible.
2. Identify and Clear the "Not NORMAL" State
The table state is likely stuck in SCHEMA_CHANGE or ROLLUP. You must cancel these before you can run REPAIR.
1. Check for active jobs:
sql
SHOW ALTER TABLE COLUMN;
SHOW ALTER TABLE ROLLUP;
SHOW ALTER TABLE OPTIMIZE;
2. Cancel any job that is not FINISHED or `CANCELLED`:
sql
CANCEL ALTER TABLE COLUMN FROM [table_name];
-- Or if it's an optimize job:
CANCEL ALTER TABLE OPTIMIZE FROM [table_name];
3. Verify the state: Once canceled, the table should return to NORMAL. Check this with SHOW TABLE [table_name].
3. Diagnose the Culprit Transaction
Check if transaction 604254 is a compaction task:
sql
SHOW TRANSACTION WHERE id = 604254;
If the TransactionType is COMPACTION, the error input rowset 8792 not found means the compaction was committed referencing a rowset that was already deleted or never existed in the object storage metadata.
4. Force Repair (Shared-Data Procedure)
Once the table is NORMAL, attempt a repair that allows for version inconsistency. This tells StarRocks to roll back to the last known good version where rowsets actually exist:
sql
ADMIN REPAIR TABLE [table_name] PROPERTIES (
"enforce_consistent_version" = "false",
"allow_empty_tablet_recovery" = "true"
);
Note: This may result in losing the data specifically from the failed transaction (604254), but it will restore the table's availability.
5. If CNs are still crashing
If CNs continue to restart, it’s often because they are trying to load the invalid metadata for tablet 423646.
* Check tablet metadata: SHOW TABLET 423646; (from your logs).
* Clear local CN cache: If a specific CN is stuck, you can try restarting it with its storage directory cleared, though in shared-data (lake) mode, the metadata usually resides in the FE's starMgr.
If the above doesn't work:
Given you are on 4.1.3, there was a known issue in earlier 3.x/4.x patch versions where dual compaction tasks on the same partition could wedge the publish queue. If the SHOW TRANSACTION result for 604254 is a compaction and you cannot clear it, the last resort is to:
1. Backup the partition (if possible).
2. ALTER TABLE [table] DROP PARTITION [partition_name] FORCE;
3. This will remove the corrupt metadata branch entirely.
References
• faq/Sql_faq.md
• administration/management/BE_configuration.md
• loading/SQL_transaction.md
• release_notes/release-3.3.md
• sql-reference/System_variable.mdDanila Fediashchin
08/10/2026, 10:08 AMlake_enable_batch_publish_version = false, now FE spams this 20 times per second:
2026-08-10 18:07:47.669+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.679+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.689+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.699+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.710+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.720+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.730+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.740+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
2026-08-10 18:07:47.750+08:00 INFO (publish-version-daemon|25) [PublishVersionDaemon.publishLakeTransactionAsync():546] start publish lake db:58879 table:133024 txn:604254
Trying dropping partition I receive an error:
The table events_u is currently in state TABLET_RESHARDRocky
08/10/2026, 10:09 AMTABLET_RESHARD state indicates that the table is undergoing an internal background process (introduced in StarRocks 4.x) to modify its bucket count or redistribute tablets. This state locks the table's metadata, which is why DROP PARTITION and ADMIN REPAIR are failing.
The infinite loop in the logs suggests that transaction 604254 is likely the "Commit" phase of this resharding job. Because it's missing a rowset, it can neither succeed nor (by default) be easily skipped.
1. Cancel the Resharding/Optimize Job
You must stop the resharding task to release the table from the TABLET_RESHARD state.
1. Find the job ID:
sql
SHOW ALTER TABLE OPTIMIZE FROM [db_name];
2. Cancel the job:
sql
CANCEL ALTER TABLE OPTIMIZE FROM [table_name];
_If this succeeds, the table state should return to NORMAL. Check with SHOW TABLE [table_name] or SHOW PROC '/dbs/[db_id]/[table_id]'._
2. Kill the Stuck Transaction
If the transaction loop continues after cancellation, manually cancel the transaction tied to the resharding/compaction:
sql
CANCEL COMPACTION WHERE TXN_ID = 604254;
(Even if it's not a standard compaction, this command often works to signal the FE to stop retrying internal maintenance transactions in shared-data clusters.)
3. Clear the Metadata Lock (Internal)
If the table remains in TABLET_RESHARD and the CANCEL command does not work, the FE's internal state machine is stuck.
1. Disable automatic resharding to prevent a new job from starting immediately:
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_auto_reshard" = "false");
2. Restart the Leader FE.
*Since you mentioned restarts didn't work before, ensuring lake_enable_auto_reshard is false before the restart is critical to prevent the Leader from immediately re-queueing the corrupt job upon boot.*
4. Recovery once State is NORMAL
Once the table is back in NORMAL state:
1. Identify the corrupt partition: The logs mentioned partition 409133.
2. Repair the table:
sql
ADMIN REPAIR TABLE [table_name] PROPERTIES("enforce_consistent_version" = "false");
3. Check for missing rowsets: If queries still fail with "rowset not found," you may need to truncate the specific partition or restore it from a backup, as the underlying files in object storage for that version chain are missing.
Why is it looping 20x/second?
In version 4.1.3, a known issue exists where certain lake (shared-data) internal tasks do not respect the standard publish_version_interval_ms backoff if they encounter a metadata inconsistency. Setting lake_enable_batch_publish_version = false isolated the failure but increased the frequency because the daemon no longer waits for other transactions to batch with it.
Recommendation: After fixing the table, upgrade to a later patch of 4.1.x when available, as these tight-loop conditions and resharding edge cases are actively being patched in the 4.1 branch.
References
• release_notes/release-2.5.md
• release_notes/release-2.0.md
• release_notes/release-2.4.mdDanila Fediashchin
08/10/2026, 10:13 AMRocky
08/10/2026, 10:14 AMTABLET_RESHARD state, combined with a COMMITTED compaction transaction that cannot be published, has created a metadata deadlock. In StarRocks 4.x, this state is typically triggered by an automatic bucket resharding or an OPTIMIZE job. Since the transaction is already committed, it is no longer "cancelable" via standard compaction commands, and the table state is preventing standard repair/drop operations.
Follow these steps to break the lock and restore the cluster:
1. Kill the Loop & Prevent New Jobs
First, stop the PublishVersionDaemon from spamming and ensure no new resharding or compaction tasks are scheduled.
sql
-- Disable automatic resharding and compaction globally
ADMIN SET FRONTEND CONFIG ("lake_enable_auto_reshard" = "false");
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
-- Keep batch publish disabled for now to isolate the failing txn
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
2. Force-Cancel the Resharding/Optimize Job
The TABLET_RESHARD state is maintained by an internal "Alter Job." You must identify and kill this job to release the table lock.
1. Check for the job:
sql
SHOW ALTER TABLE OPTIMIZE;
2. Identify the JobId for your table (events_u) and run:
sql
CANCEL ALTER TABLE OPTIMIZE WHERE JobId = [job_id];
-- OR
CANCEL ALTER TABLE OPTIMIZE FROM events_u;
3. Verify the Table State: Wait 1–2 minutes. Check if the state has returned to `NORMAL`:
sql
SHOW PROC '/dbs/58879/133024';
*Note: If the state is still TABLET_RESHARD, the FE may need a Leader restart after the lake_enable_auto_reshard config is set to false.*
3. Handle the Committed Transaction (Isolate Partition)
If the table returns to NORMAL but the log loop for transaction 604254 continues, the metadata for partition 409133 is likely corrupted.
Since you cannot cancel a committed transaction, you must remove the branch of metadata that references it.
1. Attempt to Drop the Partition:
sql
-- Using FORCE is required to bypass the version check
ALTER TABLE events_u DROP PARTITION [partition_name] FORCE;
2. If DROP fails, try to truncate it (this recreates the tablets and effectively wipes the stuck version chain):
sql
TRUNCATE TABLE events_u PARTITIONS ([partition_name]);
4. Recovery for CN Nodes
CN nodes are likely crashing because the PublishVersion task for the resharding job is asking them to load or write metadata for rowsets that don't exist.
* Once you have canceled the job or dropped the partition, restart the CN nodes.
* Check the CN <http://be.INFO|be.INFO> logs for input rowset [id] not found. If it persists for other tablets, you may need to run ADMIN REPAIR TABLE events_u PROPERTIES("enforce_consistent_version" = "false"); after the table is back in NORMAL state.
Why did this happen?
In version 4.1.3, there is a known edge case where a resharding (Optimize) task can commit a transaction referencing rowset IDs that were cleaned up by a concurrent autovacuum or compaction task. Because the resharding logic is atomic, it wedges the table until the transaction is either published (which it won't) or the metadata state is reset.
If the table remains stuck in TABLET_RESHARD after the above:
You may need to use an internal metadata tool or contact StarRocks support to manually remove the transaction from the FE's starMgr journal, as it has reached a state that the standard SQL interface cannot resolve.
References
• administration/management/FE_configuration.md
• data_source/feature-support-data-lake-analytics.md
• administration/management/compaction.mdDanila Fediashchin
08/10/2026, 10:42 AMRocky
08/10/2026, 10:42 AMTABLET_RESHARD state and the associated loop in publishLakeTransactionAsync are part of the Automatic Bucket Resharding feature (managed via OPTIMIZE jobs) in StarRocks 4.x.
The "input rowset not found" error during the publish phase is a known race condition where a compaction task and a resharding task both try to operate on the same rowsets. If compaction finishes and deletes the original rowsets before the resharding task can publish its commit, the resharding transaction enters an unrecoverable infinite loop.
Is this issue on GitHub?
There isn't a single "master issue" for this because it's a category of race conditions in the new resharding logic, but you can track related fixes and progress here:
1. #67154 (and related PRs): Fixes for "Publish Compaction crash when input rowset is not found." While your log shows it's a resharding/optimize task, the underlying "missing rowset" failure during version publishing is essentially the same root cause.
2. Search Query for GitHub: Look for recent PRs and issues with the label area/lake and keywords publishLakeTransactionAsync or reshard.
* StarRocks Lake Issues/PRs
3. Specific to 4.1.3: Since you are on a very recent build (4.1.3-8a8e186), this may be a regression or an edge case specifically in the auto-resharding logic.
How to break the loop now
Because the transaction is COMMITTED, it's essentially "written in stone" in the FE's internal StarMgr journal. To recover:
1. Disable the feature immediately:
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_auto_reshard" = "false");
2. Clear the stuck job:
Find the job in SHOW ALTER TABLE OPTIMIZE and use CANCEL ALTER TABLE OPTIMIZE WHERE JobId = ....
3. Isolate the transaction:
If the FE continues to loop after cancellation, the metadata reference is stuck in the starMgr journal. You may need to Truncate the Partition mentioned in the logs to force the FE to discard the invalid version chain and create new tablets.
sql
TRUNCATE TABLE events_u PARTITION [partition_name];
4. Restart the Leader FE:
If the state remains TABLET_RESHARD after the above, a Leader FE restart is required to refresh the in-memory state of the table after you've disabled lake_enable_auto_reshard.
Recommendation:
This behavior (looping 20x/sec) is considered a high-priority bug. I recommend opening a new issue on GitHub with your FE logs and the TABLET_RESHARD error, specifically mentioning version 4.1.3. The maintainers may provide a specific ADMIN command to manually prune the committed-but-failed transaction from the metadata store.Danila Fediashchin
08/10/2026, 1:26 PMRocky
08/10/2026, 1:26 PMlake_enable_batch_publish_version is a configuration (enabled by default since v3.2) designed for shared-data (Lake) clusters. It changes how StarRocks commits transactions to the storage engine.
How it Works
When a load job completes, it moves from a "Committed" state to "Visible." This transition requires the FE to "Publish" the version to the storage nodes.
* Disabled (false): Each transaction is published individually. FE sends an RPC to CN for every single commit.
* Enabled (true): The PublishVersionDaemon groups multiple pending transactions for the same table/partition and publishes them in a single batch RPC.
──────────
Benefits
1. Massive Ingestion Throughput: In high-concurrency scenarios (e.g., thousands of small Stream Loads per minute), batching prevents the FE from becoming an RPC bottleneck. Testing shows 5x to 10x throughput improvements for real-time ingestion.
2. Reduced Object Storage Costs: Every publish operation involves metadata writes to S3/OSS. Batching merges these metadata updates, significantly reducing the number of S3 PUT requests and associated API costs.
3. Lower FE/CN CPU Overhead: Fewer RPCs mean less context switching and lower CPU utilization on both the Frontend and Compute Nodes.
Downsides & Risks
1. Slightly Higher Visibility Latency: Transactions might wait for a few milliseconds to form a batch. If you have very low concurrency, data might take a fraction of a second longer to appear in queries.
2. Increased "Blast Radius" on Failure: If a single transaction in a batch fails to publish (e.g., due to a missing rowset or S3 timeout), the entire batch may be delayed or retried together.
3. Memory Pressure (Rare): In extremely large-scale imports with massive metadata, batching too many versions at once can occasionally lead to transient memory spikes on CN nodes (though this is rare in recent versions).
──────────
Related Parameters for Tuning
If you have lake_enable_batch_publish_version = true, you can fine-tune its behavior with:
* lake_batch_publish_max_version_num (Default: 10): The maximum number of transactions to group in one batch.
* lake_batch_publish_min_version_num (Default: 1): The minimum required to trigger a batch.
* publish_version_interval_ms (Default: 10ms): How often the daemon checks for ready transactions.
Recommendation
Keep it true for production. You should only set it to false (as you did) for *debugging purposes*—specifically to isolate a single failing transaction when your logs are being flooded, or to work around rare race conditions in specific patch versions.
References
• administration/management/FE_configuration.md
• knowledge/usage/load/shared_data_batch_ingestion.md
• sql-reference/System_variable.mdDanila Fediashchin
08/10/2026, 1:27 PMRocky
08/10/2026, 1:27 PMlake_enable_batch_publish_version does not require an FE restart.
It is a mutable configuration, meaning it can be modified at runtime using the following command:
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
How to verify the change
You can confirm the current value has been updated by running:
sql
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_enable_batch_publish_version';
Note: While the config change is instant, it only affects future publish tasks. Any publish tasks that are already in-flight or currently being processed by the PublishVersionDaemon will complete using the previous logic. In your case, since the daemon is looping on a specific stuck transaction, disabling this will help isolate that single transaction so it stops blocking other batches.
References
• administration/management/FE_configuration.md
• knowledge/usage/load/shared_data_batch_ingestion.md