http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I have a starrocks cluster version ...
# questions-and-troubleshooting
y
@Rocky I have a starrocks cluster version 4.0.10, on k8s, shred mode. At the moment, ingestion does not work for a table, and I can see the following in the log files: FE log files contains a lot of messages like this:
Copy code
2026-05-12 04:45:49.372Z INFO (PUBLISH_VERSION|25) [PublishVersionDaemon.publishLakeTransactionBatchAsync():906] start publish lake batch db:195698 table:733840 txns:763781,764987,766193,767398,767780,767823
2026-05-12 04:45:49.377Z ERROR (publish-task-16125|218677) [PublishVersionDaemon.publishPartitionBatch():703] Fail to publish partition 859787 of txnIds [763781, 764987, 766193, 767398, 767780, 767823]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[[859796, 859790, 859797, 859791, 859788, 859789]], error msg: txn log list does not contain txn log of tablet 859789, host: kube-starrocks-c
    at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:356) ~[starrocks-fe.jar:?]
    at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:392) ~[starrocks-fe.jar:?]
    at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:689) ~[starrocks-fe.jar:?]
    at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$16(PublishVersionDaemon.java:928) ~[starrocks-fe.jar:?]
    at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
    at java.lang.Thread.run(Thread.java:840) ~[?:?]
And the CN log file contains a lot of this:
Copy code
gtid: 420948406800744448
 version=22
W20260512 04:48:58.122704 139899111990848 transactions.cpp:328] Fail to get txn log: Internal error: txn log list does not contain txn log of tablet 859796 tablet_id=859796 txn=txn_id: 763781
commit_time: 1778512499
combined_txn_log: true
txn_type: TXN_NORMAL
force_publish: false
gtid: 420847444809482240
W20260512 04:48:58.122760 139899111990848 lake_service.cpp:288] Fail to publish version: Internal error: txn log list does not contain txn log of tablet 859796. tablet_id=859796 txn_ids=txn_id: 763781
commit_time: 1778512499
What is happenign and how can I fix it ?
r
The error you are encountering is related to the Shared-data (Lake) mode transaction management system, specifically the Combined Transaction Log and Aggregate Publish features introduced or enhanced in StarRocks 4.0. What is happening? 1. Inconsistency in Combined Logs: The FE is attempting to publish a batch of transactions (e.g., 
763781
). Because 
combined_txn_log
is enabled, multiple tablet updates are bundled into a single log file in object storage. 2. Missing Tablet Entry: When the CN (Compute Node) reads the transaction log for transaction 
763781
, it expects to find an entry for tablet 
859796
(and others), but the log file's internal list does not contain that tablet. 3. Publish Deadlock: Because the FE cannot successfully publish the version for these tablets, the transaction remains in the 
COMMITTED
state but never becomes 
VISIBLE
. This blocks subsequent ingestion for that table/partition because StarRocks requires versions to be published in linear order. Recommended Fixes and Workarounds 1. Disable Aggregate Publish (Immediate Workaround) The logs show the 
PublishVersionDaemon
is using the "batch" (aggregate) publish logic. Disabling this can sometimes bypass the faulty batching logic and allow transactions to be processed individually.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
2. Disable Combined Transaction Logs If the issue persists with new ingestion, disable the combined transaction log feature to prevent new transactions from being bundled in a way that might trigger this bug:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_combined_txn_log" = "false");
3. Identify and Clean the Stuck Partition If the publish continues to fail for the same transaction IDs, that specific data version is likely corrupted in the metadata. * Identify the table:
Copy code
sql
    -- Use the Table ID from your log (733840) to find the table name
    SELECT TABLE_NAME FROM information_schema.tables WHERE TABLE_ID = 733840;
* Drop the affected partition: If you can afford to re-ingest the data for that partition, dropping it is the cleanest way to clear the stuck metadata:
Copy code
sql
    ALTER TABLE <table_name> DROP PARTITION <partition_name>;
Note: After dropping, you will need to recreate the partition and reload the data. 4. Check Transaction Status Verify the state of the transactions mentioned in the error:
Copy code
sql
SHOW PROC '/transactions';
-- Look for TxnId 763781. It will likely be in the COMMITTED state.
If there are many 
COMMITTED
but not 
VISIBLE
transactions for this table, it confirms the publish queue is stuck. 5. Restart FE Leader In some cases, the FE's in-memory state regarding the "publish task" queue can become inconsistent. Restarting the FE Leader node can force a recalculation of the publish tasks. Summary of Logs * FE Error: 
Fail to publish version... txn log list does not contain txn log of tablet 859789
. This means the FE's metadata says the tablet was part of the txn, but the actual log file disagrees. * CN Error: 
Fail to get txn log... Internal error
. The CN confirms the physical log file lacks the required tablet entry. If these steps do not resolve the issue, this is likely a bug in the 4.0.x shared-data engine. It is recommended to collect the FE/CN logs and the output of 
SHOW TABLET <tablet_id>
for one of the failing tablets and reach out to the StarRocks community or support. References • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • administration/Meta_recovery.md • faq/operation_maintenance_faq.md
y
@Rocky When running:
Copy code
SELECT *
FROM information_schema.tables
WHERE TABLE_ID = 733840;
I get
Copy code
SQL Error [1064] [42000]: Getting analyzing error. Detail message: Column 'TABLE_ID' cannot be resolved.
When running
Copy code
SHOW PROC '/transactions';
I get
Copy code
10004	_statistics_
132114	digai
195697	testing
195698	dw_raw
195699	dw_staging
195700	dw_intermediate
195701	dw_marts
r
The error you got with 
information_schema.tables
is because the 
TABLE_ID
column is actually located in a different system table called 
tables_config
. 1. Identify the Table and Database Based on your logs and the 
SHOW PROC
output: * Database ID 
195698
 is 
dw_raw
. * Table ID 
733840
: Run the following query to find the table name:
Copy code
sql
    SELECT TABLE_SCHEMA, TABLE_NAME
    FROM information_schema.tables_config
    WHERE TABLE_ID = 733840;
2. Check the Stuck Transactions To see the status of the specific transactions mentioned in your log (e.g., 
763781
), you need to drill down into the database path in `SHOW PROC`:
Copy code
sql
-- Replace 195698 with your DB ID if it changes
SHOW PROC '/transactions/195698/running';
SHOW PROC '/transactions/195698/finished';
Look for the 
TransactionId
in the output. They are likely in the 
COMMITTED
state but haven't moved to 
VISIBLE
. 3. Root Cause: Shared-Data Engine Bug The error 
txn log list does not contain txn log of tablet
usually indicates a mismatch between the FE's metadata and the physical transaction logs stored in your object storage (S3/GCS/etc.). This often happens when "Combined Transaction Logs" or "Batch Publish" features are enabled in StarRocks 4.0.x. 4. How to Fix Step A: Disable the problematic features Disabling these will prevent new transactions from encountering the same metadata bundling issue and may allow the FE to recover the current publish queue.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
ADMIN SET FRONTEND CONFIG ("lake_enable_combined_txn_log" = "false");
Step B: Force a Publish Retry If the transactions remain stuck after the config change, you can try restarting the FE Leader. When the FE Leader restarts, it reloads the transaction state and retries the publish tasks. Step C: Drop and Reload (Last Resort) If the same 
table:733840
and 
partition:859787
continue to fail with the same error after the above steps, the metadata for that specific partition's latest version is likely corrupted. 1. Identify the partition name for the table found in Step 1. 2. Drop the partition: 
ALTER TABLE <table_name> DROP PARTITION <partition_name>;
3. Re-add the partition and re-ingest the data. Recommendation: Since you are on version 4.0.10 (a very new release), if disabling the configs doesn't work, this is a bug that should be reported to the community. Please provide them with the output of 
SHOW TABLET 859796;
and your FE 
fe.log
, and the results of 
SHOW PROC '/transactions/195698/finished';
. References • faq/loading/Loading_faq.md • administration/management/logs.md • knowledge/trouble_shooting/fe_deadlock_problems.md
k
This could be related to a bug we addressed recently, for now, you have to rebuild the partition to recover from the stuck transaction.
y
@Kevin Cai Thanks for the input No matter what i did, I was not able to recreate that partition. The ingestion pipe always got stuck on these issues. I created another table with all the data and that went smoothly.
k
ok
Open in Slack
PreviousNext
Powered by