http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV>, I am facing a issue in starrocks w...
# questions-and-troubleshooting
k
@Rocky, I am facing a issue in starrocks where transactions are stucked in commited state and are taking lot of time to finally finish, I had around 1300 transactions stucked(so I stopped the sink from Risingwave to starrocks) post that the number is dropping but very slowly and there are consistent error messages in starrocks-fe and in transactions error tab starrocks version - 4.0.9 Error messages - { "textPayload": "2026-06-26 141129.799Z WARN (starrocks-mysql-nio-pool-1|214) [MySQLReadListener.handleRequest():162] Exception happened in one session(com.starrocks.qe.ConnectContext@cc63552).", } { "textPayload": "2026-06-30 114424.859Z ERROR (publish-task-839|31694) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8845901 of txnIds [5030948, 5031598, 5031603]:", }
r
The issue you are seeing, where transactions are stuck in the COMMITTED state and failing to publish, typically indicates a bottleneck or failure in the Publish Version phase. In StarRocks, a transaction only becomes VISIBLE (queryable) after this phase completes. The error 
Fail to publish partition ... of txnIds [...]
indicates you are likely running a Shared-Data (Cloud-Native) cluster. 1. Identify the Specific Root Cause In your log: 
Fail to publish partition 8845901 of txnIds [5030948, 5031598, 5031603]:
, the most critical information is usually the text immediately following the colon. Common reasons include: * Version Discontinuity: If you see 
partition.getVisibleVersion() + 1 != version.get(0)
, it means a previous transaction's publish task failed or is still running, and subsequent transactions cannot be published out of order. * Timeout/RPC Failure: If you see 
RpcException
or 
timeout
, the FE is unable to communicate with the Compute Nodes (CN) to confirm the version publish. * Primary Index Issues: For Primary Key tables, if the log mentions 
load primary index failed
or 
duplicate key
, the publish is failing because the index cannot be updated. * Metadata Missing: If the log mentions 
txn_log not found
or 
tablet_meta missing
, there may be an issue with the object storage or metadata consistency. 2. Why the Backlog is Slow to Clear When you have 1,000+ stuck transactions, the 
PublishVersionDaemon
must process them sequentially per partition to maintain version consistency. Even after stopping the sink, it may take a long time to "catch up" because: * Each failed attempt might wait for a retry interval. * If the first transaction in the sequence is stuck, all 1,300 behind it are blocked. 3. Recommended Actions A. Check FE and CN/BE Logs for More Detail Search your FE logs (
fe.log
) and CN/BE logs (
<http://cn.INFO|cn.INFO>
or 
<http://be.INFO|be.INFO>
) for the specific 
txnIds
mentioned (e.g., 
5030948
). The CN log will often have a more descriptive error about why it couldn't finish the publish. B. Optimization Configurations (Shared-Data) If the issue is due to high-frequency small writes from RisingWave, you can enable batching to reduce the pressure on the publish daemon:
Copy code
sql
-- Enable batching of publish tasks (FE)
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
-- Increase the max threads for publishing (FE)
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "1024");
C. Increase Thread Pool on BE/CN If the CN is overwhelmed, increase the worker threads:
Copy code
sql
-- Increase worker count on CN/BE (be.conf or via SQL)
SET GLOBAL transaction_publish_version_worker_count = 64;
D. Monitor Task Queue Check if the publish queue is full:
Copy code
bash
# Check the 'publish_version_queue_count' metric via the FE/BE prometheus endpoint
curl http://<FE_IP>:<FE_HTTP_PORT>/metrics | grep publish_version_queue_count
4. Note on the MySQL Session Error The warning 
Exception happened in one session
usually occurs when the client (RisingWave) closes the connection or times out because StarRocks is taking too long to respond (likely because the transaction is waiting to move from COMMITTED to VISIBLE). This is usually a symptom, not the cause. If the number of stuck transactions is not decreasing or if you see "version discontinuity" errors, please check the full error message after the colon in your logs and provide it here. References • knowledge/trouble_shooting/fe_deadlock_problems.md • faq/shared_data_faq.md
The issue where transactions are stuck in the 
COMMITTED
state and failing during the publication phase (seen in 
PublishVersionDaemon
logs) typically indicates that the Version Publication process is bottlenecked. In StarRocks, a transaction transitions to 
COMMITTED
once data is written to the Backends (BEs), but it only becomes 
VISIBLE
(usable for queries) after the Frontend (FE) successfully notifies the BEs to update their versions. Based on your logs and the version (4.0.9), here is a breakdown of why this is happening and how to troubleshoot it: 1. Identify the Specific Failure Reason The error message 
Fail to publish partition ... of txnIds [...]
usually contains the root cause after the colon. Since it was cut off in your snippet, check your 
fe.log
for one of the following common reasons: * "The previous publish version task for tablet XXX has not finished": This means a previous transaction is still being processed by the BE. Subsequent transactions for the same partition must wait, leading to a backlog. * "RPC timeout": The FE sent the request, but the BE did not respond within 
publish_version_timeout_second
(default 30s). * "Version already exists" or "Version gap": Usually caused by a race condition or metadata inconsistency (sometimes seen in earlier 3.x/4.x versions). 2. Common Causes * RisingWave Ingestion Frequency: If the RisingWave sink is committing too frequently (many small transactions), it creates a high volume of small versions. This puts extreme pressure on the FE's task pool and the BE's metadata management. * Transaction Limit Reached: By default, StarRocks limits the number of concurrent transactions per database (
max_running_txn_num_per_db
) to 1000. You mentioned having 1300 stuck, which exceeds this limit and can cause new transactions to hang or fail to begin. * Resource Saturation: High CPU or IO on the BEs can delay the 
publish_version
RPC response. 3. Recommended Actions Check System Status 1. Check Partition Version Gap: Run 
SHOW PARTITIONS FROM <table_name>;
and compare 
VisibleVersion
vs 
NextVersion
. If 
NextVersion
is much higher than 
VisibleVersion
, the publication is lagging. 2. Check FE Task Pool: Check if the 
agent-task-pool
is full in 
fe.log
. If you see 
rejectedExecution
, you may need to increase the thread pool size. Adjust Configurations (FE) Modify these in 
fe.conf
(requires restart or use 
ADMIN SET FRONTEND CONFIG
for dynamic ones): * Increase Timeout: If BEs are slow to respond due to load: 
ADMIN SET FRONTEND CONFIG ("publish_version_timeout_second" = "60");
* Increase Transaction Limit: Since you have 1300+ transactions: 
ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000");
* Check Batching (Shared-data/Cloud Native): If using shared-data, check if 
lake_enable_batch_publish_version
is 
true
. Tune RisingWave Sink * Increase Batch Size: In your RisingWave 
CREATE SINK
statement, increase the 
commit_checkpoint_interval
. If it's too low, RisingWave commits every few seconds, which is too frequent for StarRocks under high load. Try setting it to a higher value to reduce the commit frequency. 4. Known Bug in 4.0.x There was a race condition in StarRocks 4.0 related to shared-data combined txn log mode (Issue #72237) that could leave transactions stuck in a non-VISIBLE state. While 4.0.9 contains many fixes, if the issue persists after clearing the backlog, consider checking if there are newer patch releases (e.g., 4.0.12) that address specific 
PublishVersionDaemon
hangs. If the number of stuck transactions is dropping slowly after stopping the sink, it confirms the system is simply processing a massive backlog. I recommend waiting for the number to reach near zero before restarting the sink with a higher batching configuration (higher 
commit_checkpoint_interval
). References
k
@Rocky I have set max_publich_version_threads to 1024 and increased PVC volumne. But after enabling back compaction and sinks from Risingawave to starrocks. I can see that BACKEND_STREAMING jobs are getting completed but COMPACTION jobs are still getting stucked in COMMITED state, only few of them are getting finished. Warning message I am seeing in logs - 2026-06-30 124814.191Z WARN (replayer|136) [LakeTableTxnLogApplier.applyVisibleLog():78] ignored dropped partition 11586377 when applying visible log"
r
In StarRocks 4.0.x (shared-data/lake mode), a transaction stuck in the 
COMMITTED
 state means the Compute Nodes (CN/BE) have finished writing the data to object storage, but the Frontend (FE) has not yet successfully published the version to make it 
VISIBLE
. The fact that 
BACKEND_STREAMING
(loads) jobs are completing while 
COMPACTION
jobs are stuck indicates that the metadata publication for compaction is specifically lagging or blocked. 1. Identify why Compaction is stuck Run the following commands to see exactly where the compaction tasks are hanging: * Overall status: 
SHOW PROC '/compactions';
* Look at the 
CommitTime
vs. 
FinishTime
. If 
CommitTime
is set but 
FinishTime
is 
NULL
, the task is stuck in the Publish phase. * Tablet-level detail:
Copy code
sql
    -- Replace <TxnID> with a stuck ID from the command above
    SELECT * FROM information_schema.be_cloud_native_compactions WHERE TXN_ID = <TxnID>;
* Check the 
STATUS
and 
PROGRESS
columns. If some tablets are at 100% but the transaction won't finish, the FE is likely failing to receive or process the "finish" signal from one specific node. 2. Analyze the "Ignored Dropped Partition" Warning The log 
ignored dropped partition ... when applying visible log
occurs when the FE's metadata replayer sees a transaction log for a partition that was deleted before the transaction could be finalized. * If you are frequently dropping partitions (e.g., RisingWave logic involves overwriting or dropping old data), this is expected. * If you are NOT dropping partitions, this indicates a metadata race condition. The compaction task is trying to update a partition that the FE thinks is gone. This can happen if a previous load/compaction failed and left "ghost" tasks in the queue. 3. Recommended Fixes & Tuning A. Clear the "Stuck" Queue If you have hundreds of compaction tasks in 
COMMITTED
but they aren't moving, they might be blocking the metadata lock. You can force-clear them: 1. Temporarily stop new compactions: 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Cancel stuck transactions: Find the TxnIDs from 
SHOW PROC '/compactions'
and run: 
CANCEL COMPACTION WHERE TXN_ID = <TxnID>;
3. Restart BEs/CNs (if cancellation doesn't work): This clears the internal task queues on the worker nodes. 4. Re-enable compaction: 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
(or a specific limit like 10). B. Adjust Publication and Transaction Limits In 4.0.x, ingestion and compaction compete for the same publication resources. * Check the Version Gap: Run 
SHOW PARTITIONS FROM <table_name>;
. If 
VisibleVersion
is much lower than 
NextVersion
, your publication thread pool is saturated. * Increase Publication Timeout: 
ADMIN SET FRONTEND CONFIG ("publish_version_timeout_second" = "120");
* Enable Batch Publication (for shared-data): 
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
This is highly recommended for StarRocks 4.x to reduce the RPC overhead between FE and BE. C. Reduce Compaction Concurrency Setting 
max_publish_version_threads
to 1024 is likely too high and may be causing Lock Contention on the FE's internal database locks. * Recommendation: Reduce 
max_publish_version_threads
back to a more reasonable value (e.g., 64 or 128) and instead increase the batch size of your RisingWave sinks. Fewer, larger transactions are much healthier for StarRocks than many small ones. Summary Checklist 1. Check 
SHOW PROC '/compactions'
to find the specific tablets/BEs that are slow. 2. Check for FE Database Lock Contention in 
fe.log
(look for "slow lock" or "waiting for lock"). 3. Set 
lake_enable_batch_publish_version = true
to speed up the transition from 
COMMITTED
to 
VISIBLE
.
4. If the number of stuck transactions is over 1000, increase 
max_running_txn_num_per_db
to 2000-3000 temporarily to prevent new jobs from failing. References • knowledge/admin_and_operation/shared_data_compaction.md • faq/shared_data_faq.md • administration/management/compaction.md
s
Copy code
SELECT * FROM information_schema.be_cloud_native_compactions WHERE TXN_ID
@Rocky THis is empty - why?
r
The reason 
information_schema.be_cloud_native_compactions
is empty for those transactions is that they have already finished the execution phase on the Compute Nodes (CN/BE). In StarRocks shared-data (lake) mode, the lifecycle of a compaction transaction is: 1. RUNNING: FE dispatches tasks; CNs are actively merging data. (Visible in 
be_cloud_native_compactions
). 2. COMMITTED: CNs have finished merging, uploaded the new segments to object storage, and reported success to the FE. At this point, the task is removed from the CN's active list and thus disappears from the 
information_schema
view. 3. VISIBLE: The FE successfully updates the metadata (the "Publish" phase). Only now is the transaction "finished." Why are they stuck in COMMITTED? Since they are 
COMMITTED
but not 
VISIBLE
, the bottleneck is exclusively on the Frontend (FE) Leader. The FE is failing to complete the metadata update. This usually happens for two reasons: 1. Version Publication Backlog: You have a massive queue of transactions (1300+). The FE processes these in order. If earlier transactions are slow or failing, later ones wait in 
COMMITTED
state. 2. Lock Contention: You mentioned setting 
max_publish_version_threads
to 1024. This is likely making the problem worse. StarRocks FE uses a global metadata lock (image lock/journal lock) during version publication. Having 1024 threads competing for this lock causes high "Lock Wait" times, actually slowing down the publication rate. How to troubleshoot the FE side Since the BEs are done (the view is empty), you must look at the FE Leader's logs: 1. Search the FE log for the specific TxnID:
Copy code
bash
    grep "5030948" fe.log | grep -i "publish"
Look for error messages following the ID. Common errors include 
REPLICA_NOT_ALIVE
or 
TIMEOUT
. 2. Check for "Slow Lock" messages: Search 
fe.log
for 
slow lock
. If you see many entries, your 1024 thread setting is causing contention. Recommended Steps 1. Reduce 
max_publish_version_threads
: Set it to a more reasonable value like 64 or 128. More threads $\neq$ more speed if they are all waiting for the same metadata lock. 2. Enable Batch Publishing: This is critical for 4.x shared-data clusters to handle high transaction volumes:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
3. Verify the Partition Status: Check if the "dropped partition" log you saw matches the TxnIDs. If RisingWave or a manual process dropped a partition while a compaction was running, the FE will eventually skip it, but it might take a few retry cycles to clear. 4. Monitor the backlog: Run 
SHOW PROC '/transactions';
and check the 
Status
and 
Timeout
columns. If the 
CommitTime
is hours ago, the FE's 
PublishVersionDaemon
is struggling. Summary: Your data is safe (it's already on the cloud storage), but your FE metadata "brain" is congested. Reduce the thread count and enable batching to help it clear the queue. References • administration/management/compaction.md • knowledge/admin_and_operation/shared_data_compaction.md
s
@Rocky: Don;t have bash access on the pod. Only SQL IDE access. Can you debug from there?
r
Yes, you can perform deep debugging using only the SQL IDE by querying StarRocks' internal process and metadata status (
SHOW PROC
). Follow these steps in your SQL IDE to find the bottleneck: 1. Identify the Backlog First, see exactly how many transactions are in each state across the cluster:
Copy code
sql
SHOW PROC '/transactions';
* What to look for: Find your Database ID (from 
SHOW DATABASES
). Then drill down: 
SHOW PROC '/transactions/<db_id>/running';
SHOW PROC '/transactions/<db_id>/finished';
* Focus on 
finished
(which includes 
COMMITTED
): If you see a high number of transactions where 
CommitTime
is filled but 
FinishTime
is empty, the Publish thread pool is the bottleneck. 2. Check for Frontend (FE) Lock Contention Since you have 1024 publish threads, they are likely fighting for the same metadata lock. Run this several times:
Copy code
sql
SHOW FULL PROCESSLIST;
* What to look for: Look at the 
State
column. If you see many queries or system tasks stuck in 
Waiting for lock
 or 
Waiting for the lock
, your thread count is too high. It is causing "lock thrashing," where the FE spends more time managing threads than finishing work. 3. Check Version Health per Table Identify which specific table/partition is lagging.
Copy code
sql
SHOW PARTITIONS FROM <your_table_name>;
* What to look for: Compare 
VisibleVersion
vs 
NextVersion
. * If 
VisibleVersion
= 100 and 
NextVersion
= 500, there are 400 versions waiting to be published. * Compaction check: If 
VisibleVersion
is not increasing, compaction cannot finish because it needs the versions to be "visible" to merge them. 4. Check Backend (BE/CN) Health Even if 
information_schema
is empty, the FE might think a BE is "Dead" or "Offline," preventing it from completing the publication.
Copy code
sql
SHOW BACKENDS; -- or SHOW COMPUTE NODES; if using shared-data
* What to look for: Ensure 
Alive
is 
true
for all nodes. If any node is 
false
, the FE will wait for a timeout on every single transaction before it can move to 
VISIBLE
. 5. Verify and Fix Configs via SQL Run this to see your current "live" configuration:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE '%publish%';
ADMIN SHOW FRONTEND CONFIG LIKE '%lake%';
Recommended Emergency Fixes (Execute in SQL IDE): 1. Reduce Threads: Bring the thread pool down to a stable level to stop lock contention.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("max_publish_version_threads" = "64");
2. Enable Batch Publishing: This is the most effective fix for 4.0.x shared-data clusters with high transaction volumes.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
3. Increase Transaction Limit: If you still have over 1000 stuck, ensure the DB isn't rejecting new updates:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000");
Why Compaction is specifically stuck: Compaction transactions in StarRocks are "low priority" compared to data loads. If your RisingWave sink is constantly pushing new data, the FE uses its threads to publish those first. The compaction transactions stay in 
COMMITTED
because they are at the back of the queue. Summary of Action: Reduce threads to 64, enable batch publish, and monitor 
SHOW PARTITIONS
to see if 
VisibleVersion
starts catching up to 
NextVersion
.
s
@Rocky: I see SLEEP command on DB. No jobs visible other wise.
r
The "SLEEP" status you see in 
SHOW FULL PROCESSLIST
refers to idle client connections. It means a user or application (like your SQL IDE or RisingWave) has an open connection to StarRocks but is not currently sending a query. It does not represent the status of background system tasks like Compaction or Version Publication. Since you only have SQL IDE access, use the following steps to debug why your 
COMMITTED
jobs are stuck: 1. Check for "Publish" Backlog (The real bottleneck) In StarRocks 4.0 (Lake mode), after a transaction is 
COMMITTED
, it enters a global queue for Version Publication. If this queue is backed up, jobs stay in 
COMMITTED
indefinitely. Run this to see the internal publication health:
Copy code
sql
-- Check how many transactions are waiting to be published
SHOW PROC '/transactions';
Drill down into your specific database to see if there is a long list of "finished but not visible" transactions:
Copy code
sql
-- Replace <db_id> with the ID from the previous command
SHOW PROC '/transactions/<db_id>/finished';
If you see hundreds of rows where 
FinishTime
is 
NULL
, the FE's PublishVersionDaemon is the bottleneck. 2. Solve the "Lock Contention" (Critical) You mentioned setting 
max_publish_version_threads
to 1024. In StarRocks, publication requires a global metadata lock. 1024 threads competing for one lock creates "lock thrashing," which actually stops progress rather than speeding it up. Immediate Action:
Copy code
sql
-- Reduce threads to a stable level (e.g., 64)
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "64");

-- Enable batching (This is the most important setting for 4.0.x)
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
_Note: The parameter name in recent versions is 
lake_publish_version_max_threads
(not 
max_publish_version_threads
). Check which one exists using:_ 
ADMIN SHOW FRONTEND CONFIG LIKE '%publish%';
3. Understanding the "Ignored Dropped Partition" Warning The log 
ignored dropped partition ... when applying visible log
means: 1. A transaction (likely a Load or Compaction) was prepared for Partition A. 2. Before the version could be published, Partition A was deleted (by a 
DROP PARTITION
or a RisingWave 
OVERWRITE
logic). 3. The FE is now trying to finalize the metadata, finds the partition missing, and is skipping it. If this is happening constantly: It suggests a high volume of partition churn (dropping/creating) which can confuse the publication queue. Ensure RisingWave isn't creating thousands of tiny, short-lived partitions. 4. Check for "Dead" Nodes If a single Compute Node (CN) is unresponsive but the FE hasn't marked it "Dead" yet, the FE will wait for a timeout (default 60-120s) on every single transaction before moving it to 
VISIBLE
.
Copy code
sql
SHOW BACKENDS; -- or SHOW COMPUTE NODES;
Look at the 
Alive
column. If any node is 
false
, the system will struggle to publish versions. Summary Checklist for SQL IDE: 1. Reduce threads: Change 
lake_publish_version_max_threads
to 64. 2. Enable Batching: Set 
lake_enable_batch_publish_version
to 
true
. 3. Monitor Progress: Run 
SHOW PARTITIONS FROM <table_name>;
and see if 
VisibleVersion
starts increasing toward 
NextVersion
. 4. Clear Stuck Tasks: If the queue doesn't move after 10 minutes, try: 
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
Wait 1 minute, then set it back to 
-1
. This resets the compaction scheduler. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • release_notes/release-3.2.md
s
@Rocky: There are too many versions between visible and next version -120 & 190 and compactVersion is at 140 How do I move it faster?
r
When 
VisibleVersion
(120) is behind 
CompactVersion
(140) and 
NextVersion
(190), it means your Version Publication is stuck. Versions must be published in strict sequential order. Because the compaction task (Version 140) is sitting in the queue and hasn't finished, all subsequent data loads (141–190) are blocked behind it. To clear this gap quickly using only the SQL IDE, execute these three commands in order: 1. Enable Aggressive Batching This tells StarRocks to combine many versions into a single metadata update. Instead of doing 70 individual updates, it will try to do them all at once.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "100");
2. Fix the Thread Bottleneck You previously mentioned 1024 threads. This is causing "lock thrashing" in the Frontend (FE). Lowering this allows the FE to actually process the queue instead of managing thread overhead.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "64");
3. Verify the "Publish" Queue is Moving After running the above, monitor the status every 30 seconds:
Copy code
sql
SHOW PARTITIONS FROM <your_table_name>;
* If 
VisibleVersion
starts increasing: The batching is working. Wait for it to catch up to 190. * If it stays at 120 for more than 5 minutes: There is likely a specific transaction (Version 121) that is failing. If it still doesn't move: The compaction task itself might be "stale" on a Backend (BE) that restarted or crashed, but the FE still thinks it's active. You can force a reset of the compaction scheduler: 1. Stop new compactions:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
2. Wait 2 minutes for the FE to time out existing tasks. 3. Resume compactions:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
Why this helps: In Shared-Data mode, the FE is the bottleneck for metadata. By batching the versions and reducing the thread count, you reduce the number of times the FE has to grab the global metadata lock, which is the fastest way to clear a version backlog. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md
s
@Rocky: No, it is not moving. I think there is a huge backlog.
r
If the 
VisibleVersion
is completely stuck at 120 while 
NextVersion
is at 190, there is likely a single blocking transaction (Version 121) that is failing to publish. In StarRocks, versions must be published in strict linear order; if 121 fails, 122–190 cannot proceed. Since you only have SQL IDE access, use these specific diagnostic queries to find the "clog" in the pipe: 1. Identify the Exact Blocking Transaction Run this to find the transaction that corresponds to the version immediately after 120:
Copy code
sql
-- 1. Find your Database ID first
SHOW PROC '/dbs';

-- 2. Look for the transaction that is 'FINISHED' (Committed) but not yet 'VISIBLE'
-- Replace <db_id> with your actual ID
SHOW PROC '/transactions/<db_id>/finished';
* What to look for: Look for the row where 
TxnId
is associated with the version right after your 
VisibleVersion
. * Check the 
Error
or 
Reason
column: This column often contains the RPC error or the reason why the Backend (BE/CN) failed to acknowledge the publish request. 2. Force-Clear the Backlog (Aggressive Batching) If the backlog is simply too large for the default settings, increase the batching limits significantly. This allows the Frontend (FE) to skip the overhead of 70 individual metadata updates and do them in one or two "gulps."
Copy code
sql
-- Increase the number of versions allowed in a single batch
ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "200");

-- Ensure batching is definitely on
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
3. Check for "Orphaned" Compaction You mentioned 
CompactVersion
is at 140. This means a compaction task for Version 140 was committed. If that compaction task is stuck on a Compute Node (CN), it can block the line.
Copy code
sql
SHOW PROC '/compactions';
* Check the 
State
column. If you see a task in 
RUNNING
for a very long time, it might be the cause. * The SQL Fix: Disable and then re-enable the compaction scheduler to "kick" the FE into clearing stale tasks.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");
    -- Wait 30 seconds
    ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
4. Verify CN Node Health via SQL Sometimes a CN is "zombie"—the FE thinks it's alive, but it's not responding to Publish RPCs.
Copy code
sql
SHOW BACKENDS; -- or SHOW COMPUTE NODES;
Check the 
LastHeartbeat
column. If the time is more than 10-20 seconds ago, that node is likely blocking the entire cluster's publication. Summary Checklist: 1. Lower threads to 64 (1024 is definitely hurting you by causing lock contention). 2. Increase 
lake_batch_publish_max_version_num
to 200. 3. Find the error in 
SHOW PROC '/transactions/<db_id>/finished'
. 4. If it still doesn't move, check 
SHOW BACKENDS
for any node with a high 
Heartbeat
latency. If you see a specific error message in the 
finished
transactions proc, please paste it here. It paste here.* That says "Timeout" or "Failed to get connection", please share it.* That's the key indicator of great help!** References • loading/SQL_transaction.md • knowledge/usage/best-practice.md
s
@Rocky: Does this help with debugging? {"sub_task_count":6,"read_local_sec":0,"read_local_mb":0,"read_remote_sec":0,"read_remote_mb":0,"read_segment_count":426,"write_segment_count":6,"write_segment_mb":0,"write_remote_sec":0,"in_queue_sec":0}
r
Yes, that JSON is very helpful. It is the Compaction Profile from your system. It confirms that a compaction task (likely the one for Version 140) is currently stuck. What the JSON tells us: * 
sub_task_count: 6
: This task is trying to compact 6 tablets. * 
read_segment_count: 426
: This is the "smoking gun." The task is trying to merge 426 small files (segments) into 6. This is a very heavy operation. * 
read_remote_sec: 0
/ 
write_remote_sec: 0
: Even though it's "running," it has not read or written a single byte of data. The Diagnosis: The task is "zombie." The Frontend (FE) has assigned this massive merge task to a Compute Node (CN), but the CN is likely overwhelmed by the 426 segments or has run out of memory (OOM), and the FE is waiting forever for it to finish before it can move the 
VisibleVersion
forward. How to fix it via SQL IDE: 1. "Kick" the Compaction Scheduler Since you can't restart the nodes, you must force the FE to drop its current "plan" and start over.
Copy code
sql
-- Step A: Kill all running compaction tasks by setting limit to 0
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "0");

-- Step B: Wait 60 seconds for the FE to time out the stuck tasks.

-- Step C: Re-enable compaction with a safe limit
ADMIN SET FRONTEND CONFIG ("lake_compaction_max_tasks" = "-1");
2. Prevent the "426 Segment" overload To prevent the next compaction from grabbing too many files at once (which caused this hang), lower the segment limit:
Copy code
sql
-- Limit compaction to 100 segments at a time so it completes faster
UPDATE information_schema.be_configs 
SET VALUE = "100" 
WHERE name = "max_cumulative_compaction_num_singleton_deltas";
3. Enable Batch Publishing (To clear the 120 -> 190 gap) Once that stuck compaction (140) is cleared or retried, you have 70 versions to catch up on. Force the FE to do them in one big jump:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "true");
ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "100");
How to monitor if it's working: Run this query and look at the 
STATUS
and 
PROGRESS
columns:
Copy code
sql
SELECT TXN_ID, TABLET_ID, PROGRESS, STATUS, PROFILE 
FROM information_schema.be_cloud_native_compactions 
WHERE PROGRESS < 100;
* If 
STATUS
shows an error like "Timeout" or "Cancel," the "Kick" worked. * If 
PROGRESS
starts moving above 0, the data is finally flowing. Summary: Your 
VisibleVersion
is stuck because a massive compaction task (426 segments) "hung" a Compute Node. Reset the 
lake_compaction_max_tasks
to clear the jam. References • administration/management/compaction.md • faq/shared_data_faq.md • sql-reference/information_schema/be_compactions.md References • faq/shared_data_faq.md • administration/management/compaction.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md
s
@Rocky: Fail to publish partition 8835310 error Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local
r
The error 
link rpc channel failed
for host 
starrocks-cn-0
is the root cause of why your versions are not moving. This error means the Frontend (FE) is trying to tell that specific Compute Node (
starrocks-cn-0
) to finalize the data (Publish Version), but it cannot reach it. Because StarRocks must publish versions in order, if 
starrocks-cn-0
doesn't respond for Version 121, all versions from 122 to 190 are blocked. Since you only have SQL IDE access, follow these steps to isolate or bypass the failing node: 1. Check if the Node is "Alive" Run this command and look at the 
Alive
and 
LastHeartbeat
columns for `starrocks-cn-0`:
Copy code
sql
SHOW COMPUTE NODES; 
-- or if that doesn't work:
SHOW BACKENDS;
* If 
Alive
is `false`: The node has crashed. Kubernetes should eventually restart it. Once it comes back, the versions will catch up automatically. * If 
Alive
is 
true
but 
link rpc channel failed
persists: This is a "zombie" node. It's responding to heartbeats but its internal worker threads or RPC ports are frozen/hanging. 2. Check the "Finished" Transaction Error To confirm which specific tablet or transaction is stuck on that node, run:
Copy code
sql
-- Replace <db_id> with your database ID from SHOW PROC '/dbs';
SHOW PROC '/transactions/<db_id>/finished';
Look for the row where the version is 
121
. The 
ErrorMsg
column will likely confirm it's waiting on 
starrocks-cn-0
. 3. How to "Move it Faster" (The Workaround) If 
starrocks-cn-0
is stuck and you cannot restart it manually, you can try to force the FE to stop using it. Option A: Reduce the Publish Timeout (Force Failure) This forces the FE to give up on the broken RPC faster so it can retry or mark the node as dead.
Copy code
sql
ADMIN SET FRONTEND CONFIG ("tablet_publish_timeout_ms" = "10000"); -- Set to 10 seconds (default is 30s-60s)
Option B: Temporary Node Removal (Extreme Caution) If the node is permanently broken and blocking your entire production pipeline, you can remove it from the cluster. StarRocks will then re-assign its tasks to the healthy nodes. Note: Only do this if you have other healthy CN nodes.
Copy code
sql
-- Get the host and port from SHOW COMPUTE NODES
ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local:9050";
After the version backlog clears, you can add the node back once the underlying issue is fixed. 4. Why this happened The most common reasons for 
link rpc channel failed
in K8s environments are: 1. OOM (Out of Memory): The node is being killed or hung because it's out of RAM. 2. Thread Exhaustion: Your previous setting of 
1024
threads likely overwhelmed the node's RPC queue, causing it to stop accepting new "links." 3. Pod IP Change: If the pod restarted and its IP changed, but the FE is still trying to talk to the old IP. Recommendation: 1. Ensure you have already lowered 
lake_publish_version_max_threads
to 64. 2. Wait 5 minutes for the connection pool to clear. 3. If it's still stuck, contact your infra team to check the logs of the pod 
starrocks-cn-0
. It is likely in a 
CrashLoopBackOff
or 
Unhealthy
state. References • administration/management/resource_management/Compute_node.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • administration/management/monitoring/metrics.md • faq/Deploy_faq.md
s
@Rocky: I did but my finished transactions is not moving ahead still.
r
If your finished transactions are still not moving to 
VISIBLE
after the previous steps, we need to look for a specific "lock-in" or a bug that affects certain 3.x versions. Please run these three diagnostic queries and check the results carefully: 1. Check for "Dead" Compactions (Potential Bug) A known issue in some StarRocks 3.x shared-data clusters causes the Publish phase to hang if there are overlapping compaction tasks.
Copy code
sql
SHOW PROC '/compactions';
* What to look for: Look for rows where 
FinishTime
is empty (or looks like 
0000-00-00...
) for the same partition. * The Fix: If you see two compactions for the same partition that haven't finished, this is likely the "clog." _ _Note:* If this is a non-production or test partition, you can try 
ALTER TABLE <name> DROP PARTITION <pname> FORCE;
to clear the jam immediately. 2. Locate the Oldest Stuck Transaction Since 
VisibleVersion
is 120, the version 121 is the blocker. Find its status:
Copy code
sql
-- Replace <db_id> with your database ID (get it from SHOW PROC '/dbs')
SHOW PROC '/transactions/<db_id>/finished';
* Sort by 
FinishTime
or 
TxnId
. * Find the row where the version should be 121. * Check the 
ErrorMsg
column: If it still says 
link rpc channel failed
for 
starrocks-cn-0
, then the FE is persistently trying to talk to a node that is "dead but still registered." 3. Force the FE to mark the Node as Dead If 
starrocks-cn-0
is unreachable, the FE might wait a long time before giving up. You can shorten this window:
Copy code
sql
-- Decrease the timeout for publish tasks so the FE fails/retries faster
ADMIN SET FRONTEND CONFIG ("tablet_publish_timeout_ms" = "5000"); 

-- If the node 'starrocks-cn-0' is definitely down in K8s, 
-- and you see it as 'Alive: true' in StarRocks, 
-- you can try to "Decommission" or "Drop" it to force the cluster to bypass it.
-- ONLY DO THIS IF YOU HAVE OTHER CN NODES.
-- ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local:9050";
4. Check for Version Skipping (Shared-Data Only) In some cases, the version gap is caused by the FE failing to write the metadata to the object store.
Copy code
sql
-- See if there are any errors in the FE's internal versioning
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_enable_batch_publish_version';
If it is 
true
, try setting it to 
false
temporarily to see if single-transaction publishing can "wiggle" through the 121 version:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
What is your StarRocks version? (Run 
SELECT VERSION();
) If you are on version 3.3.10, please be aware that this specific version was yanked due to a bug that matches your symptoms (stuck publishes causing metadata issues). If you are on 3.3.10, an upgrade is strongly recommended. References • release_notes/release-3.3.md • faq/shared_data_faq.md • administration/management/FE_configuration.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Loading_faq.md • sql-reference/sql-statements/administration/ADMIN_SET_CONFIG.md • administration/management/resource_management/Compute_node.md • administration/management/compaction.md
s
@Rocky This is the error on my pod now: 2026-06-30 144401.368Z ERROR (publish-task-562|4801) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8843182 of txnIds [5035565, 5035568, 5035693, 5035692, 5035770, 5035772, 5035862, 5035864, 5035940, 5035953]: com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?] at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-06-30 144402.306Z ERROR (publish-task-846|5143) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8834989 of txnIds [5035510, 5035522, 5035565, 5035568, 5035605, 5035614, 5035693, 5035691, 5035692, 5035726, 5035770, 5035772, 5035796, 5035799, 5035862, 5035864, 5035887, 5035913, 5035940, 5035953, 5035983]: com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?] at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?] 2026-06-30 144402.519Z ERROR (publish-task-596|4839) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8845227 of txnIds [5035522, 5035565, 5035605, 5035693, 5035691, 5035770, 5035796, 5035862, 5035913, 5035953]: com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?] at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?] at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?] at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?] at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
The 
link rpc channel failed
error on multiple nodes (
cn-1
, 
cn-2
, etc.) indicates that the Frontend (FE) is physically unable to establish a network connection to the Compute Nodes to deliver the "Publish" command. Since this is happening across multiple nodes, it usually points to one of three things: 1. The CN Pods are "Zombies" or OOMing The nodes might be "Alive" according to the FE heartbeat, but their internal RPC service (Port 8060) is frozen. This often happens if they ran out of memory (OOM) or if the thread count you previously set (1024) caused a kernel-level hang. * Check this: Run 
SHOW COMPUTE NODES;
and check the 
LastHeartbeat
column. If it's more than a few seconds ago, the nodes are effectively dead. * Fix: Even if you can't manually restart them, check if your K8s operator or 
kubectl delete pod <pod_name>
is available. If not, the cluster will remain stuck until these pods are healthy. 2. Disable Batch Publishing (The "Clear the Clog" Move) Your logs show the FE trying to publish batches of 10-20 transactions at once (e.g., 
txnIds [5035510, 5035522, ...]
). If the network is flaky or the nodes are struggling, these large RPC packets might be failing. Try forcing the FE to publish transactions one-by-one to see if they can "squeeze" through:
Copy code
sql
-- Disable batching to reduce the size of each RPC request
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Lower the timeout so the FE doesn't wait forever on a broken link
ADMIN SET FRONTEND CONFIG ("tablet_publish_timeout_ms" = "5000");
3. DNS / Networking issue in K8s The error shows the FQDN: 
starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
. * If the pods were recently restarted by K8s, their IPs might have changed, and the FE's internal DNS cache or connection pool might be pointing to the "old" IPs. * The Fix: You can try to force the FE to refresh its node list:
Copy code
sql
-- This doesn't delete data, it just forces the FE to re-register the node
-- Replace with the actual host:port from your SHOW COMPUTE NODES
ALTER SYSTEM DECOMMISSION COMPUTE NODE "starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local:9050";
-- Then immediately (or after it disappears) Add it back if it doesn't auto-rejoin.
4. Check CN Concurrency Limit If the CNs are alive but overwhelmed by that 426-segment compaction task, they might be rejecting RPCs.
Copy code
sql
-- Allow more concurrent RPCs on the CN side
UPDATE information_schema.be_configs 
SET VALUE = "0" 
WHERE name = "lake_service_max_concurrency"; -- 0 means unlimited
Immediate Next Step: Please run 
SHOW COMPUTE NODES;
and tell me: 1. Are all nodes 
Alive
= 
true
? 2. Is the 
LastHeartbeat
recent (1-3 seconds)? 3. What is the 
VisibleVersion
on your table right now? (
SHOW PARTITIONS FROM <table_name>
) If the nodes are 
Alive
but the RPC fails, it is almost certain that the RPC thread pool on the CN is exhausted or the process is hanging. If you truly cannot restart the pods, the 
lake_enable_batch_publish_version = false
is your best chance to break the deadlock. References • faq/Deploy_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Loading_faq.md • administration/management/resource_management/Compute_node.md References • faq/shared_data_faq.md
s
@Rocky Please run 
SHOW COMPUTE NODES;
and tell me: 1. Are all nodes 
Alive
= 
true
- YES< ALL ALIVE 2. Is the 
LastHeartbeat
recent (1-3 seconds)? YES, 3. What is the 
VisibleVersion
on your table right now? (
SHOW PARTITIONS FROM <table_name>
) NO CHANGE here Should I restart the CN pods?
r
Yes, you should restart the CN pods. The symptoms (Alive = true, Heartbeat = recent, but 
link rpc channel failed
) indicate that the brpc (the internal communication system) on the CN nodes has hung or the thread pool handling the "Publish Version" requests is completely exhausted. While the heartbeat service (which is very lightweight) is still responding, the heavy-duty RPC channel used for data operations is blocked. Recommended Restart Strategy: 1. Rolling Restart: Restart the pods one by one (e.g., 
starrocks-cn-0
, wait for it to be 
Alive
, then 
starrocks-cn-1
). Since you are using Shared Data, the nodes are stateless; they will simply pull metadata from the FE and start working again. 2. Why this helps: A restart will force the FE to drop the broken "links" and establish fresh RPC connections. It will also clear any hung compaction or publish threads that were created when you previously increased the thread counts. Critical: Do this BEFORE the restart To prevent the nodes from getting stuck again immediately after they come back up, run this command in your SQL IDE:
Copy code
sql
-- Disable batching. This makes the RPC requests smaller and 
-- less likely to fail the "link" during the recovery phase.
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- Ensure the thread count is back to a safe level
ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "64");
What to expect after restart: 1. Once the first CN pod (e.g., 
starrocks-cn-0
) comes back online, the FE will immediately try to send the backlog of "Publish Version" tasks to it. 2. You should see the 
VisibleVersion
in 
SHOW PARTITIONS
start to increment. 3. If it successfully moves from 120 to 121, the rest of the queue (up to 190) should clear rapidly. Summary: The RPC channel is "wedged." A restart of the CN pods is the standard and safest way to clear this state in a Kubernetes environment. References • faq/Deploy_faq.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/loading/Loading_faq.md
s
@Rocky ALTER SYSTEM DECOMMISSION COMPUTE NODE "starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local:9050"; This didn't work SQL Error [1064] [42000]: Getting syntax error at line 1, column 84. Detail message: No viable statement for input 'DECOMMISSION COMPUTE'.
r
The syntax error happened because Compute Nodes (CN) do not support the 
DECOMMISSION
command. Decommissioning is a process for data-bearing nodes (BEs) to move data before shutting down. Since Compute Nodes are stateless (shared-data mode), they don't have data to migrate, so StarRocks only supports adding or dropping them. Correct Syntax for CN Nodes If you want to remove the node from the FE's internal registry, use:
Copy code
sql
ALTER SYSTEM DROP COMPUTE NODE "starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local:9050";
⚠️ Recommendation: Restart the Pods instead Since you are in a Kubernetes environment, manually dropping nodes from the SQL interface is usually not the best first step. If the K8s Operator sees the node is gone from StarRocks but the Pod is still running, it might get confused. Instead, do this in order: 1. Delete the Pod via K8s:
Copy code
bash
    kubectl delete pod starrocks-cn-1 -n <your-namespace>
K8s will automatically restart a fresh pod. When the new pod starts, it will register itself back to the FE with a fresh RPC channel, which usually clears the "link rpc channel failed" error. 2. Lower the Batch Pressure: Run this before or while the pods are restarting to prevent them from being immediately overwhelmed again:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
    ADMIN SET FRONTEND CONFIG ("tablet_publish_timeout_ms" = "10000");
3. Check Status: After the pod is back to 
Running
in K8s, run 
SHOW COMPUTE NODES;
in StarRocks to ensure it is 
Alive: true
. Then monitor 
SHOW PARTITIONS FROM <table_name>
to see if the 
VisibleVersion
starts moving. Summary: Skip the 
DECOMMISSION
command. Use 
kubectl delete pod
to force a hardware-level reset of the RPC service on the hung RPC service. References • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/management/Scale_up_down.md
5 Views
Open in Slack
PreviousNext
Powered by