http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV>: Getting this error for multiple pa...
# questions-and-troubleshooting
s
@Rocky: Getting this error for multiple partition
Copy code
2026-06-30 14:47:32.048Z ERROR (publish-task-749|5021) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8904723 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
The error 
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed
indicates a network connectivity failure between the Front End (FE) and a Compute Node (CN) in your StarRocks Shared-data (Lake) cluster. Specifically, the FE failed to establish or maintain a bRPC (the internal communication protocol) connection to the host 
starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
while attempting to batch-publish transaction versions. Common Causes 1. CN Node Unavailability: The pod 
starrocks-cn-1
may have crashed, restarted, or is currently in a 
NotReady
state. 2. DNS or Network Issues: The FE cannot resolve or reach the CN's internal Kubernetes DNS name. 3. Resource Exhaustion: If the CN is under heavy load (high CPU/Memory), it may stop responding to RPC heartbeats, causing the FE to drop the "link." 4. High Concurrency: The FE is attempting to publish too many versions simultaneously, exceeding the RPC connection or thread limits. Recommended Troubleshooting Steps 1. Check CN Pod Health Verify the status of the specific CN node mentioned in the error:
Copy code
bash
kubectl get pods -n <your-namespace> | grep starrocks-cn-1
Check the logs of that CN pod for any crashes or OOM (Out of Memory) errors:
Copy code
bash
kubectl logs starrocks-cn-1 -n <your-namespace> --tail=100
2. Verify Network Connectivity Exec into your FE pod and try to reach the CN on its RPC port (default is 9060):
Copy code
bash
# Inside FE pod
nc -zv starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local 9060
If this fails, there is a network policy, DNS issue, or the CN process is not listening. 3. Review FE Configuration In Shared-data mode, StarRocks batches publish tasks. You can check these parameters in your FE (via 
admin show frontend config;
): * 
lake_publish_version_max_threads
: (Default 512) The maximum threads for publishing. If your system is extremely busy, you may need to monitor FE thread usage. * 
lake_enable_batch_publish_version
: (Default true) This is enabled in your logs (
publishPartitionBatch
). It reduces RPCs by grouping transactions. If you suspect batching is causing timeouts due to size, you can temporarily disable it to see if single-transaction publishes succeed, though this is usually less efficient. 4. Check for Version Accumulation If you see 
multiple partition
errors, your cluster might be falling behind on publishing. Check if there are many "Visible" but not yet "Finished" transactions:
Copy code
sql
SHOW BACKENDS; -- Check if any CNs are "Alive" = 'false'
SHOW PROC '/transactions'; -- Check for a backlog of transactions
If the CN pod is healthy and the network is fine, restarting the FE (or the leader FE) can sometimes clear stale RPC channels, but prioritize checking the CN logs first to ensure it isn't crashing under load. References • faq/loading/Loading_faq.md • faq/Deploy_faq.md • administration/management/BE_configuration.md • faq/operation_maintenance_faq.md • faq/loading/Routine_load_faq.md • administration/Meta_recovery.md
s
@Rocky I restarted the pods on your suggestion and these errors are back2026-
Copy code
06-30 15:09:29.289Z ERROR (publish-task-184|4321) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8902872 of txnIds [5035590, 5035634, 5035657, 5035717, 5035817, 5035838, 5035859, 5035926, 5035929, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-06-30 15:09:29.521Z ERROR (publish-task-887|5188) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8901848 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-06-30 15:09:31.501Z ERROR (publish-task-113|4234) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8899813 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-06-30 15:09:32.009Z ERROR (publish-task-917|5220) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8904932 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
r
Since the issue persisted after a restart and is affecting multiple Compute Nodes (
starrocks-cn-1
, 
starrocks-cn-2
), the "link rpc channel failed" error suggests a systemic communication problem between the FE and the CNs. In StarRocks Shared-data (Lake) mode, the FE batches multiple transaction publishes into a single RPC call to optimize performance. Your logs show batches of 21 transactions at once, which might be exceeding RPC limits or timing out. Recommended Steps 1. Reduce or Disable Publish Batching As a primary diagnostic step, disable the batch publishing feature to see if single-transaction publishes succeed. If the "link" failure is caused by a payload that is too large or complex for the network/RPC buffer, this will bypass it. Run these on your Leader FE:
Copy code
sql
-- Disable batching temporarily
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");

-- OR, keep batching but reduce the max versions per batch (default is 10)
ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "5");
If this resolves the error, your network or CNs were likely struggling with the large batch payloads. 2. Check for Service Mesh (Istio/Linkerd) If you are running in Kubernetes with a service mesh like Istio: * StarRocks uses bRPC for internal communication. bRPC connections are long-lived. * Service meshes sometimes terminate idle or long-running bRPC/HTTP2 connections aggressively, leading to "link rpc channel failed." * Action: Try bypassing the sidecar for StarRocks internal ports (9060 for CN/BE) or disable the mesh for the StarRocks namespace to see if the errors stop. 3. Verify DNS and MTU The error uses a very specific FQDN: 
starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
. * DNS Latency: If your K8s DNS (CoreDNS) is slow, the FE might fail to "link" within the RPC connection timeout. * MTU Issues: If you are using a VPN or a specific CNI (like Calico/Cilium) with a lower MTU (e.g., 1450), large RPC packets (common in batch publishes) might be dropped. * Test: Try to 
ping
the CN from the FE pod with a large packet size: 
ping -s 1472 starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
4. CN Side Resource Check If the CN is under extreme CPU pressure or is performing heavy I/O (e.g., during high-concurrency ingestion), the bRPC worker threads might be saturated, causing it to drop new connection attempts. * Check the CN logs (
<http://be.INFO|be.INFO>
) for messages like 
Reached the maximum number of connections
or 
brpc
related warnings. * Check if the CN pods are hitting their CPU Limits in Kubernetes (
kubectl top pods
). 5. Check Transaction Backlog The fact that it's trying to publish 21 transactions at once for a single partition suggests a backlog.
Copy code
sql
-- Check if transactions are accumulating in "VISIBLE" state but not finishing
SHOW PROC '/transactions';
Summary of action: If you need an immediate fix to get data moving, set 
lake_enable_batch_publish_version = false
. If the error continues even with single-transaction publishes, it is almost certainly a Kubernetes network/DNS issue between the pods. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md
s
@Rocky Nope, didn't get resolved
Copy code
2026-06-30 15:13:12.474Z ERROR (publish-task-890|5192) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8904984 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-06-30 15:13:13.642Z ERROR (publish-task-474|4692) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8902949 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972, 5036348]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-06-30 15:13:13.912Z ERROR (publish-task-416|4619) [PublishVersionDaemon.publishPartitionBatch():690] Fail to publish partition 8899877 of txnIds [5035590, 5035602, 5035634, 5035637, 5035657, 5035697, 5035717, 5035737, 5035776, 5035781, 5035817, 5035838, 5035848, 5035859, 5035861, 5035900, 5035918, 5035926, 5035929, 5035960, 5035972]:
com.starrocks.rpc.RpcException: Fail to publish version for tablets:[], error msg: link rpc channel failed, host: starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local
	at com.starrocks.lake.Utils.sendAggregatePublishVersionRequest(Utils.java:347) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.Utils.aggregatePublishVersion(Utils.java:362) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.publishPartitionBatch(PublishVersionDaemon.java:676) ~[starrocks-fe.jar:?]
	at com.starrocks.transaction.PublishVersionDaemon.lambda$publishLakeTransactionBatchAsync$15(PublishVersionDaemon.java:818) ~[starrocks-fe.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
Also, CPU usage of my FE pods is 0.36 whereas restricted is at 12
r
The fact that the error persists across multiple nodes (
cn-0
, 
cn-1
, 
cn-2
) and continues to attempt publishing large batches (21+ transactions) suggests that the communication layer is failing to handle the requests. Since your FE CPU is low and you've already tried restarting, the issue is likely at the Networking/RPC layer or Configuration synchronization. 1. Check Backend/CN Registration Run the following command in your StarRocks SQL client:
Copy code
sql
SHOW COMPUTE NODES; 
-- or if you use standard BEs
SHOW BACKENDS;
* Check the 
IP
and 
Hostname
columns: Do they match the 
starrocks-cn-X.starrocks-cn-search...
addresses in the error? * Check 
Alive
status: If they are 
false
, the FE has already marked them as unreachable. * Check 
BrpcPort
: Ensure it is actually 9060 (the default). If your Kubernetes service or container uses a different port, the "link" will fail. 2. Force-Disable Batching on the LEADER The logs still show 
PublishVersionDaemon.publishPartitionBatch
and a list of 21 
txnIds
. This means batching is still active. In a Shared-data cluster, if one transaction in a batch causes a connection glitch, the whole batch fails and retries, creating a loop. Please verify you are connected to the Leader FE and run:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
-- Also reduce the timeout to fail fast and retry individual txns
ADMIN SET FRONTEND CONFIG ("lake_publish_version_timeout_ms" = "10000");
Wait 30 seconds and check the 
fe.log
. The log lines should change from 
publishPartitionBatch
to 
publishVersionForLakeTable
(singular transaction). 3. Kubernetes Network Policy / MTU If you are using a CNI like Calico or Cilium, or if you are on a Cloud Provider (AWS/GCP/Azure) with specific MTU settings: * Large RPC packets (which happen when you batch 20+ transactions) can exceed the MTU and get dropped without an ICMP response, leading to a "link failure." * Test: From the FE pod, try to reach a CN using a large ping packet:
Copy code
bash
    ping -s 1400 starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local
If this fails but a small ping (
ping -s 64 ...
) succeeds, you have an MTU mismatch in your cluster network. 4. Check for 
brpc
Errors in CN Logs Even though the FE says "link failed," the CN might be rejecting the connection. Check the 
<http://be.INFO|be.INFO>
(or 
<http://cn.INFO|cn.INFO>
) logs on one of the CN pods: * Look for: 
too many open files
* Look for: 
Reached max connection
* Look for: 
brpc
or 
ServiceThreadPool
exhaustion. 5. Check "trino.svc" Hostname Resolution The hostname 
starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local
is slightly unusual (referencing a 
trino
service/namespace). * Ensure that from the FE pod, 
nslookup
or 
getent hosts
correctly resolves that address to the Pod IP, not the Service ClusterIP. StarRocks requires direct Pod-to-Pod communication for bRPC. * If your 
fe.conf
or 
cn.conf
has 
BE_IP
or 
priority_networks
set incorrectly, the node might be registering an address that is not routable from the FE. Next Step Recommendation: If setting 
lake_enable_batch_publish_version = false
still results in logs showing 
publishPartitionBatch
with multiple IDs, the configuration didn't stick. Use 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake%';
to confirm the setting. References • administration/management/FE_configuration.md • faq/loading/Loading_faq.md • faq/shared_data_faq.md • faq/Deploy_faq.md
s
@Rocky Check 
BrpcPort
: Ensure it is actually 9060 (the default). If your Kubernetes service or container uses a different port, the "link" will fail. Mine is 9060 - how do we check if that's what it is supposed to be?
r
In StarRocks, the Front End (FE) communicates with Compute Nodes (CN) using two primary ports. It is very likely that you are looking at the Thrift Port (9060) while the error is occurring on the bRPC Port (8060). 1. Distinguish the Ports * Thrift Port (
be_port
): Default is 9060. This is used for basic control tasks and heartbeat. If 
SHOW COMPUTE NODES
says the node is "Alive," the FE successfully connected to 9060. * bRPC Port (
brpc_port
): Default is 8060. This is used for the Lake Service (publish version) and data transmission. The "link rpc channel failed" error refers to this port. 2. How to check what the FE is using Run this command in your SQL client:
Copy code
sql
SHOW COMPUTE NODES\G
-- or
SHOW BACKENDS\G
Look for the following columns: * 
IP
/ 
Host
: Ensure this is the actual Pod IP or a resolvable internal DNS name. * 
BrpcPort
: This should be 8060 by default. If this says 9060, your configuration is wrong (you cannot run bRPC and Thrift on the same port). 3. Verify on the CN Pod Exec into one of your CN pods and check if the bRPC port is actually listening:
Copy code
bash
# Check if 8060 is listening
netstat -tunlp | grep 8060

# Check your configuration file
grep "brpc_port" /path/to/starrocks/conf/cn.conf
_If 
brpc_port
is not set in 
cn.conf
, it defaults to 8060._ 4. Kubernetes Service Check Since you are using Kubernetes (
trino.svc.cluster.local
), check your Service and Pod definitions: * Service: Ensure your Kubernetes Service (likely named 
starrocks-cn-search
) has port 8060 exposed. * Pod: Ensure the container has port 8060 opened. * Network Policy: If you have a NetworkPolicy, it must allow traffic from the FE pod to the CN pods on port 8060. Why did it work before? If this was working and suddenly stopped: 1. Port Exhaustion: If there are too many concurrent connections, the CN might stop accepting new bRPC "links." 2. FQDN Issues: The FE is trying to connect to 
starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local
. If the Kubernetes DNS resolution is slow or failing, the bRPC client will fail to "link" the channel before the timeout. Recommendation: Try to 
telnet
or 
nc
from the FE pod to a CN pod specifically on port 8060:
Copy code
bash
nc -zv starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local 8060
If this times out or is refused, but port 9060 works, you have found the cause: the bRPC port is blocked or not running. References • deployment/environment_configurations.md • administration/management/BE_configuration.md • faq/Deploy_faq.md • administration/management/FE_configuration.md • faq/loading/Loading_faq.md References • deployment/environment_configurations.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/Deploy_faq.md
s
@Rocky: It was working before and has suddenly stopped working for a few transactions. Published, but they have not been finished. These are the IPs: starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local starrocks-cn-1.starrocks-cn-search.trino.svc.cluster.local starrocks-cn-2.starrocks-cn-search.trino.svc.cluster.local
r
The fact that your logs still show 
publishPartitionBatch
with 21 
txnIds
means the command to disable batching did not take effect. In StarRocks, configuration changes must be applied to the Leader FE. If you run the command on a Follower FE, it will appear to succeed in the console but won't change the cluster's behavior. 1. Verify and Fix Config (Leader FE) Check which FE is the leader and apply the change there: 1. Run 
SHOW FRONTENDS;
and find the one where 
IsMaster
(or 
IsLeader
) is 
true
. 2. Connect your SQL client directly to that IP/Host. 3. Run the commands again:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
    ADMIN SET FRONTEND CONFIG ("lake_batch_publish_max_version_num" = "1");
4. Verify with: 
ADMIN SHOW FRONTEND CONFIG LIKE 'lake_enable_batch_publish_version';
2. The Port 8060 vs 9060 Issue You mentioned your port is 9060. That is the Thrift Port. The error 
link rpc channel failed
happens on the bRPC Port, which is 8060 by default. * The Problem: If your Kubernetes Service or NetworkPolicy only allows port 9060, the "Heartbeat" (which uses 9060) will work, and the node will show as "Alive," but the "Publish Version" (which uses 8060) will fail with a "link failed" error. * The Fix: Ensure port 8060 is open in your Kubernetes Service (
starrocks-cn-search
) and that your pods can communicate over it. 3. Check for "Overcrowded" CNs If the bRPC port is open but the connection still fails, the CN might be rejecting it. Look into the 
<http://be.INFO|be.INFO>
(or 
<http://cn.INFO|cn.INFO>
) logs on 
starrocks-cn-0/1/2
for this specific string: 
The server is overcrowded
or 
Reached max connection
If you see "overcrowded" errors, add this to your 
cn.conf
and restart the CNs:
Copy code
conf
brpc_socket_max_unwritten_bytes = 4294967296
4. DNS / Namespace mismatch Your hostnames end in 
trino.svc.cluster.local
. * Is your StarRocks cluster actually running in a namespace named 
trino
? * If your FEs and CNs are in different namespaces, K8s might be blocking the bRPC traffic. * Test connection from FE pod: 
nc -zv starrocks-cn-0.starrocks-cn-search.trino.svc.cluster.local 8060
If this says "Connection refused" or "Timeout," your network/K8s service is the bottleneck. Summary The FE is failing to "link" to the CNs on port 8060. Because batching is still on, one single connection failure is failing 21+ transactions at once. 1. Force-disable batching on the Leader FE. 2. Ensure port 8060 is accessible between pods. References • administration/management/FE_configuration.md • knowledge/usage/load/shared_data_batch_ingestion.md • faq/shared_data_faq.md
s
@Rocky: BRPC port is 8060 - false alarm there But these partitions are still failing. Can I skip over them somehow?
r
Yes, you can "skip" or force-fix these partitions, but because StarRocks (shared-data) relies on a strict sequential versioning system, you have to do this by either rolling back the partition to a known good state or manually forcing the version forward. Option 1: Rollback to a Good Version (Safest) If you can afford to lose the data from the transactions that are currently failing to publish, you can use 
ADMIN REPAIR
. This attempts to revert the partition metadata to the last successful version in object storage.
Copy code
sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>) 
PROPERTIES ("enforce_consistent_version" = "true");
* Result: This will "unstick" the partition by dropping the failing transactions and resetting the 
VisibleVersion
to the last healthy one. You will then need to re-load the data that was lost. Option 2: Manually Advance the Version (High Risk) If you know exactly what the "healthy" version should be (you can see this from 
SHOW PARTITIONS FROM <table_name>
), you can try to force the metadata forward. Warning: This can lead to data corruption if the data files for that version don't actually exist in S3/MinIO.
Copy code
sql
ADMIN SET TABLE <table_name> PARTITION (<partition_name>) VERSION TO <target_version>;
_ _Note: You must run this while connected directly to the Leader FE.* Option 3: Drop and Recreate (Nuclear Option) If the partition is completely corrupted and you just want to start over for that specific time range: 1. Drop the partition: 
ALTER TABLE <table_name> DROP PARTITION <partition_name> FORCE;
2. Wait 1 minute for metadata to clear. 3. Re-insert the data. This will create a fresh partition with a new ID, bypassing the old stuck version sequence. Why you are still seeing batches (Action Required) You mentioned the batching didn't stop. If you see 
publishPartitionBatch
with 21 IDs in the logs, the config change did not work. This happens if: 1. You weren't on the Leader: Run 
SHOW FRONTENDS
and ensure 
IsLeader
is 
true
for your current session. 2. The daemon was already "in-flight": The daemon may have already queued those 21 transactions. Try this to force-clear the queue: 1. Identify the Leader FE. 2. Run:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("lake_enable_batch_publish_version" = "false");
    ADMIN SET FRONTEND CONFIG ("lake_publish_version_max_threads" = "64"); -- Lower concurrency to reduce RPC pressure
3. Restart the Leader FE. This is the only way to hard-reset the 
PublishVersionDaemon
task queue if it's stuck in a loop. Final Verification After performing one of the "skip" options above, run this to see if the version sequence is moving again:
Copy code
sql
SHOW PROC '/transactions';
-- Look for transactions in 'COMMITTED' state. If the number is decreasing, it's working.
Recommendation: Try Option 1 (REPAIR) first. It is the built-in way to handle "stuck" versions in cloud-native tables without manually editing metadata hacking the metadata manually hacking the metadata. References • loading/SQL_transaction.md • administration/management/compaction.md • administration/Data_recovery.md • faq/Others.md • loading/Stream_Load_transaction_interface.md • release_notes/release-2.5.md
s
@Rocky How do I know which partition this belongs to?
r
To find which partition a transaction (by 
txnId
) belongs to, the most direct way is to query the internal metadata tables. Method 1: Using SQL (Recommended) If you are on StarRocks 3.1.9+ or 3.2.4+, you can join the transaction metadata with the partition metadata:
Copy code
sql
SELECT 
    p.DB_NAME, 
    p.TABLE_NAME, 
    p.PARTITION_NAME, 
    t.TXN_ID
FROM information_schema.be_txns t
JOIN information_schema.partitions_meta p 
    ON t.PARTITION_ID = p.PARTITION_ID
WHERE t.TXN_ID = <your_txn_id>;
Method 2: Check via Transaction Label If you don't have the 
partitions_meta
table yet, you can find the "Label" of the transaction, which often contains the table name or can be traced back to the load job: 1. Find the Label:
Copy code
sql
    SHOW TRANSACTION WHERE ID = <your_txn_id>;
* Look for the 
Label
column. 2. Find the Load Job: If the Label looks like a Stream Load or Broker Load (e.g., 
my_load_job_123
), you can find the table via:
Copy code
sql
    SHOW LOAD WHERE Label = '<the_label_you_found>';
* The 
JobDetails
column in 
SHOW LOAD
will list the target table. Method 3: Using 
SHOW PROC
 This is the manual way to browse the metadata hierarchy: 1. Run 
SHOW PROC '/transactions';
to get the 
DbId
of your database. 2. Run 
SHOW PROC '/transactions/<DbId>';
to see the transaction list. _ _Note: This view is often summarized. Method 1 is much more efficient for a specific ID.* Summary of what to do next: Once you have the Table Name and Partition Name: 1. Run 
SHOW PARTITIONS FROM <table_name> WHERE PartitionName = '<partition_name>';
2. Note the 
VisibleVersion
and 
NextVersion
. 3. If 
NextVersion
is much higher than 
VisibleVersion
and it's not moving, proceed with the 
ADMIN REPAIR
command mentioned previously.
Copy code
sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>) 
PROPERTIES ("enforce_consistent_version" = "true");
References • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • sql-reference/information_schema/be_txns.md • knowledge/admin_and_operation/shared_data_compaction.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md References • knowledge/admin_and_operation/shared_data_compaction.md
s
@Rocky The results of this query is null
Copy code
SELECT
    p.DB_NAME,
    p.TABLE_NAME,
    p.PARTITION_NAME,
    t.TXN_ID
FROM information_schema.be_txns t
JOIN information_schema.partitions_meta p
    ON t.PARTITION_ID = p.PARTITION_ID
WHERE t.TXN_ID in (5035565, 5035693, 5035770, 5035862, 5035953);
r
The reason your 
information_schema.be_txns
query returned 
NULL
is that this table typically only shows "active" transaction tasks currently being handled by BE/CN nodes. For transactions that have already been Committed but are stuck in the Publish phase, the record exists primarily in the FE's memory and the global transaction manager. Here are the definitive ways to identify the table and partition for those specific IDs: Method 1: The 
SHOW TRANSACTION
Command This is the most reliable way to find the metadata for a specific transaction ID. You must run this for each ID:
Copy code
sql
SHOW TRANSACTION WHERE ID = 5035565;
Look for the Label column in the output. * Routine Load: The label will contain the job name. * Stream Load: The label is usually provided by your loading client. * Insert Into: The label looks like 
insert_...
. Once you have the label, you can find the table by running:
Copy code
sql
SHOW LOAD WHERE Label = '<Label_from_above>';
The 
JobDetails
or 
TableNames
column will tell you which table it belongs to. Method 2: FE Log Search (Fastest for many IDs) Since you have the IDs, you can grep the FE 
fe.audit.log
or 
fe.log
on the Leader FE. This is often faster than running individual SQL commands.
Copy code
bash
grep -E "5035565|5035693|5035770|5035862|5035953" fe.audit.log
The audit log entry for the 
COMMIT
or 
LOAD
will explicitly list the database and table being modified. Method 3: 
SHOW PROC
(The "Manual" Drill-down) If the above don't work, you can find which database contains the stuck transactions: 1. Run 
SHOW PROC '/transactions';
. This lists 
DbId
and 
DbName
. 2. For each 
DbId
(e.g., 10005), run: 
SHOW PROC '/transactions/10005';
. 3. This shows the count of transactions in 
running
, 
finished
, etc. 4. If a database has many 
committed
but not 
finished
transactions, that is your target. You can then run:
Copy code
sql
    SHOW PROC '/transactions/10005/running'; -- or 'finished'
This will list the transactions along with their 
TableIdList
. How to unstick them once you find the table Once you identify the 
<table_name>
, check the partition status:
Copy code
sql
SHOW PARTITIONS FROM <table_name>;
Check if 
VisibleVersion
is lagging behind 
NextVersion
. If it is, and you've already verified the BRPC port (8060) is open, use the repair command on that specific partition:
Copy code
sql
ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>) 
PROPERTIES ("enforce_consistent_version" = "true");
Note: If these are all for the same table, it is highly likely that one "older" transaction failed to publish, and all subsequent transactions (the ones you listed) are queued behind it because StarRocks publishes versions in strict linear order. Finding that "first" stuck transaction is the key. References • faq/shared_data_faq.md • administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md
s
@Rocky SHOW LOAD WHERE Label = 'rw-txn-c53af80f-123e-48d7-a283-019de3e43c77-1782823713310007'; This is also null. So I don't know which table
r
The 
rw-txn
prefix is a signature of the StarRocks Flink Connector (specifically when using the Two-Phase Commit / Exactly-Once mode). Because these are handled as "Stream Load" transactions rather than standard Broker/Insert loads, they don't always appear in 
SHOW LOAD
. To find the table, use these steps in order: 1. The Global Transaction Lookup Instead of 
SHOW LOAD
, use the generic transaction viewer. You must run this for one of your specific IDs:
Copy code
sql
SHOW TRANSACTION WHERE ID = 5035565;
* If this returns empty: It means the transaction is in a different database than your current session. * How to find the database: Run 
SHOW PROC '/transactions';
. This will show you a list of all databases (
DbId
and 
DbName
) and how many transactions are currently 
COMMITTED
(waiting to publish). Note the 
DbName
that has a high number in the 
COMMITTED
column. 2. Query the metadata for that Database Once you know the database name (e.g., 
analytics_db
), run the check again with the database context:
Copy code
sql
SHOW TRANSACTION FROM analytics_db WHERE ID = 5035565;
The output will have a Label. Since it's a Flink transaction, the label itself won't name the table, but the 
TableIdList
 or the logs will. 3. Cross-reference Table ID to Name If 
SHOW TRANSACTION
gives you a 
TableIdList
(e.g., 
12345
), you can find the name with:
Copy code
sql
SELECT TABLE_NAME 
FROM information_schema.tables_config 
WHERE TABLE_ID = 12345;
Why 
information_schema.be_txns
was null That view only shows "active" data-writing tasks currently on the BEs. Once a transaction is Committed (but stuck in Publish), the data is already on the BEs/S3, and the transaction lives in the FE Leader's memory waiting to be finalized. Summary Action Plan 1. Run 
SHOW PROC '/transactions';
to identify the database with the "stuck" counts. 2. Run 
SHOW TRANSACTION FROM <db_name> WHERE ID = 5035565;
to confirm it exists there. 3. Run 
SHOW PARTITIONS FROM <likely_table_name>;
for the tables in that database. Look for the one where 
VisibleVersion
is stuck and much lower than 
NextVersion
. 4. Apply the 
ADMIN REPAIR
command to that partition. Note on Flink: If these are Flink transactions and you "skip" them or repair the partition, Flink might throw an error on its next checkpoint because the transaction it expected to commit was manually interfered with. You may need to restart the Flink job from a fresh checkpoint afterward. References • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • sql-reference/information_schema/be_txns.md • loading/SQL_transaction.md • sql-reference/sql-statements/loading_unloading/INSERT.md • sql-reference/System_variable.md • administration/Data_recovery.md • faq/Others.md • loading/Stream_Load_transaction_interface.md • release_notes/release-2.5.md • knowledge/admin_and_operation/shared_data_compaction.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md
s
@Rocky, there are close to 100 tables in this DB. I don't want to play a guessing game here.
r
To avoid searching 100 tables manually, you can use the system metadata to pinpoint exactly where the bottleneck is. Since these transactions are likely "stuck" in the publish phase (Committed but not yet Visible), they won't appear in 
be_txns
. Instead, they are waiting for a specific partition to advance its version. 1. Scan all 100 tables for the "Stuck" Partition Run this query to find any table where the versions are not advancing. This will identify the table and partition these IDs belong to:
Copy code
sql
SELECT 
    DB_NAME, 
    TABLE_NAME, 
    PARTITION_NAME, 
    VISIBLE_VERSION, 
    NEXT_VERSION,
    (NEXT_VERSION - VISIBLE_VERSION) AS version_gap
FROM information_schema.partitions_meta
WHERE DB_NAME = '<your_db_name>'
  AND NEXT_VERSION > VISIBLE_VERSION + 1
ORDER BY version_gap DESC;
* Why this works: If your transactions (5035565, etc.) are queued, the 
NEXT_VERSION
will be significantly higher than the 
VISIBLE_VERSION
. This query scans all 100+ tables in seconds and points you to the culprit. 2. Verify the Transaction ID via 
SHOW TRANSACTION
 
SHOW LOAD
often returns null for Flink/Stream Load transactions because they are handled differently. Use this specific command instead:
Copy code
sql
-- Replace <db_name> with your database name
SHOW TRANSACTION FROM <db_name> WHERE ID = 5035565;
In the output, look for the 
TableIdList
 column. If it provides an ID (e.g., 
12345
), you can get the table name immediately with:
Copy code
sql
SELECT TABLE_NAME FROM information_schema.tables_config WHERE TABLE_ID = 12345;
3. Why is it stuck? Since these are Flink 
rw-txn
transactions, the most common reasons for them to stay in a "Committed" state (stuck) are: 1. Version Dependency: A previous transaction (an ID smaller than yours) failed to publish. In StarRocks, versions must be published in strict order. If transaction 
5035560
is stuck, your 
5035565
cannot finish. 2. BE Connection: The FE cannot communicate with a BE to confirm the write (Check if Port 8060/9050 is open). Solution Once you identify the 
<table_name>
and 
<partition_name>
from the query in Step 1: 1. Repair the partition:
Copy code
sql
    ADMIN REPAIR TABLE <table_name> PARTITION (<partition_name>)
    PROPERTIES ("enforce_consistent_version" = "true");
2. Wait 1 minute. The stuck transactions should automatically move from 
COMMITTED
to 
VISIBLE
. References • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • faq/shared_data_faq.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • knowledge/admin_and_operation/shared_data_compaction.md References • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • sql-reference/information_schema/partitions_meta.md • administration/management/compaction.md
2 Views
Open in Slack
PreviousNext
Powered by