http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV>, We are currently evaluating the s...
# questions-and-troubleshooting
v
@Rocky, We are currently evaluating the starrocks for our use case overall it is working good for our use case but we were using single FE till now. but we want to DR testing for our use case and for that purpose we have added 2 follower FE's and used the recommended proxysql to do the loadbalancing. when proxySQL is querying the follower FE we are seeing frequent rpc failures. I am sharing following sql output:
Copy code
MySQL [helpdesk_system]> SHOW frontends;
+------+----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| Id   | Name                             | IP            | EditLogPort | HttpPort | QueryPort | RpcPort | Role     | ClusterId  | Join | Alive | ReplayedJournalId | LastHeartbeat       | IsHelper | ErrMsg | StartTime           | Version       |
+------+----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
| 2    | 10.99.111.118_9010_1782404017953 | 10.99.111.118 | 9010        | 8030     | 9030      | 9020    | LEADER   | 1811725254 | true | true  | 7865664           | 2026-07-15 21:57:37 | true     |        | 2026-06-26 00:13:44 | 4.1.1-14b7e3f |
| 3    | 10.99.111.112_9010_1784116515938 | 10.99.111.112 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 1811725254 | true | true  | 7865662           | 2026-07-15 21:57:37 | true     |        | 2026-07-15 20:11:42 | 4.1.1-14b7e3f |
| 4    | 10.99.111.102_9010_1784116528639 | 10.99.111.102 | 9010        | 8030     | 9030      | 9020    | FOLLOWER | 1811725254 | true | true  | 7865662           | 2026-07-15 21:57:37 | true     |        | 2026-07-15 20:15:52 | 4.1.1-14b7e3f |
+------+----------------------------------+---------------+-------------+----------+-----------+---------+----------+------------+------+-------+-------------------+---------------------+----------+--------+---------------------+---------------+
3 rows in set (0.012 sec)

MySQL [helpdesk_system]> SELECT COUNT(*) as c FROM helpdesk_system.flexifields;
ERROR 1064 (HY000): rpc failed with 10.99.111.108: exec rpc error. backend [id=80554] [host=10.99.111.108]
MySQL [helpdesk_system]> SELECT COUNT(*) as c FROM helpdesk_system.flexifields;
ERROR 1064 (HY000): rpc failed with 10.99.111.91: exec rpc error. backend [id=10021] [host=10.99.111.91]
MySQL [helpdesk_system]> SELECT COUNT(*) as c FROM helpdesk_system.flexifields;
ERROR 1064 (HY000): rpc failed with 10.99.111.83: exec rpc error. backend [id=10002] [host=10.99.111.83] 
MySQL [helpdesk_system]> show backends;
+-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
| BackendId | IP            | HeartbeatPort | BePort | HttpPort | BrpcPort | LastStartTime       | LastHeartbeat       | Alive | SystemDecommissioned | ClusterDecommissioned | TabletNum | DataUsedCapacity | AvailCapacity | TotalCapacity | UsedPct | MaxDiskUsedPct | ErrMsg | Version       | Status                                                 | DataTotalCapacity | DataUsedPct | CpuCores | MemLimit | NumRunningQueries | MemUsedPct | CpuUsedPct | DataCacheMetrics                                         | Location | StatusCode |
+-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
| 10006     | 10.99.111.103 | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:22:36 | 2026-07-15 22:18:12 | true  | false                | false                 | 158       | 231.667 GB       | 625.059 GB    | 884.324 GB    | 29.32 % | 29.32 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:02"} | 856.727 GB        | 27.04 %     | 16       | 50.038GB | 1                 | 31.49 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/190GB, MemUsage: 10GB/10GB |          | OK         |
| 80554     | 10.99.111.108 | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:18:46 | 2026-07-15 22:18:12 | true  | false                | false                 | 163       | 195.026 GB       | 662.076 GB    | 884.324 GB    | 25.13 % | 25.13 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:03"} | 857.102 GB        | 22.75 %     | 16       | 50.038GB | 0                 | 30.39 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/150GB, MemUsage: 10GB/10GB |          | OK         |
| 80521     | 10.99.111.111 | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:31:56 | 2026-07-15 22:18:12 | true  | false                | false                 | 166       | 194.840 GB       | 663.476 GB    | 884.324 GB    | 24.97 % | 24.97 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:03"} | 858.317 GB        | 22.70 %     | 16       | 50.038GB | 0                 | 31.04 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/220GB, MemUsage: 10GB/10GB |          | OK         |
| 10023     | 10.99.111.74  | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:31:51 | 2026-07-15 22:18:12 | true  | false                | false                 | 149       | 226.739 GB       | 630.787 GB    | 884.324 GB    | 28.67 % | 28.67 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:02"} | 857.526 GB        | 26.44 %     | 16       | 50.038GB | 0                 | 30.86 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/200GB, MemUsage: 10GB/10GB |          | OK         |
| 10002     | 10.99.111.83  | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:21:06 | 2026-07-15 22:18:12 | true  | false                | false                 | 152       | 230.066 GB       | 626.791 GB    | 884.324 GB    | 29.12 % | 29.12 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:02"} | 856.857 GB        | 26.85 %     | 16       | 50.038GB | 0                 | 32.29 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/200GB, MemUsage: 10GB/10GB |          | OK         |
| 10021     | 10.99.111.91  | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:24:06 | 2026-07-15 22:18:12 | true  | false                | false                 | 151       | 238.067 GB       | 618.579 GB    | 884.324 GB    | 30.05 % | 30.05 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:02"} | 856.645 GB        | 27.79 %     | 16       | 50.038GB | 0                 | 31.22 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/170GB, MemUsage: 10GB/10GB |          | OK         |
| 10022     | 10.99.111.93  | 9050          | 9060   | 8040     | 8060     | 2026-07-02 20:26:01 | 2026-07-15 22:18:12 | true  | false                | false                 | 159       | 239.884 GB       | 618.215 GB    | 884.324 GB    | 30.09 % | 30.09 %        |        | 4.1.1-14b7e3f | {"lastSuccessReportTabletsTime":"2026-07-15 22:18:02"} | 858.100 GB        | 27.96 %     | 16       | 50.038GB | 0                 | 30.63 %    | 0.0 %      | Status: Normal, DiskUsage: 0B/160GB, MemUsage: 10GB/10GB |          | OK         |
+-----------+---------------+---------------+--------+----------+----------+---------------------+---------------------+-------+----------------------+-----------------------+-----------+------------------+---------------+---------------+---------+----------------+--------+---------------+--------------------------------------------------------+-------------------+-------------+----------+----------+-------------------+------------+------------+----------------------------------------------------------+----------+------------+
7 rows in set (0.004 sec)
Can you share do we need to update any settings ?
r
Based on the error message 
rpc failed with [BE IP]: exec rpc error
, it appears that your Follower FE nodes are unable to communicate with the Backends (BEs) over the RPC network, even though the cluster metadata (controlled by the Leader) shows all nodes as "Alive." In StarRocks, while the Leader FE handles heartbeats and metadata updates, every FE (Leader or Follower) must have direct network access to every BE to send query execution plans. 1. Check Network Connectivity (Port 8060) The error 
exec rpc error
specifically points to failures in the bRPC (internal RPC) communication. By default, BEs listen on port 8060 (
brpc_port
) for these requests. From each of your Follower FE nodes (
10.99.111.112
and 
10.99.111.102
), verify they can reach the BEs on port 8060:
Copy code
bash
# Run this from the Follower FE machines
telnet 10.99.111.108 8060
telnet 10.99.111.91 8060
telnet 10.99.111.83 8060
If these connections time out or are refused, you likely have a firewall rule or security group blocking communication between these specific FE and BE subnets/IPs. 2. Verify 
priority_networks
 If your machines have multiple network interfaces (e.g., public and private IPs), StarRocks might be binding to an interface that isn't reachable by the other nodes. * Ensure that 
priority_networks
is configured in both 
fe.conf
and 
be.conf
on all nodes. * It should be set to the CIDR of the internal network you want them to use (e.g., 
priority_networks = 10.99.111.0/24
). * If this was not set, the BE might be reporting an IP address to the Leader that the Followers cannot reach. 3. Check FE Logs for Details On the Follower FE that is failing the query, check the 
fe.warn.log
or 
fe.log
. Look for a more detailed stack trace following the 
exec rpc error
. Common underlying causes found in the logs include: * Connection refused/timeout: Confirms network/firewall issues. * Packet size limit: If you are querying very large schemas, you might need to increase 
brpc_max_body_size
(though this is rare for a simple 
COUNT(*)
). 4. ProxySQL and Follower Read Consistency Since you are using ProxySQL to load balance across Followers: * Metadata Lag: Follower FEs may have a slight lag in replaying metadata journals compared to the Leader. In your 
SHOW FRONTENDS
output, the 
ReplayedJournalId
for your followers (
7865662
) is slightly behind the Leader (
7865664
). This is normal and usually shouldn't cause an RPC error, but if the lag grows significantly, queries might fail. * Consistency: If you require strict session consistency (reading your own writes immediately), ensure ProxySQL is configured to route those specific queries to the Leader FE, as Followers might not have the latest data yet. Summary of Recommended Actions 1. Firewall: Ensure port 8060 (brpc_port) and 9060 (be_port) are open from all FEs to all BEs. 2. Config: Ensure 
priority_networks
is consistent across the cluster. 3. Logs: Check 
fe.warn.log
on the Follower FE for the specific "Reason" (e.g., 
Connection refused
or 
Socket timeout
). If the network is verified and the issue persists, try running the query directly against the Leader FE. If it works there but fails on all Followers, it confirms the issue is specific to the Follower-to-BE network path. References • administration/management/BE_configuration.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md • faq/loading/Loading_faq.md • administration/management/monitoring/metrics.md
v
@Rocky, attaching the logs: fe.warn logs: 2026-07-15 153136.623Z WARN (thrift-server-pool-860|1173) [DefaultCoordinator.updateRuntimeProfile():1277] exec state report failed status=errorCode CANCELLED InternalError, query_id=019f6666-bb8a-7527-88c8-904e53c1c64f, instance_id=019f6666-bb8a-7527-88c8-904e53c1c650, backend_id=10021 2026-07-15 153136.623Z WARN (thrift-server-pool-860|1173) [DefaultCoordinator.updateStatus():920] one instance report fail throw updateStatus(), need cancel. job id: -1, query id: 019f6666-bb8a-7527-88c8-904e53c1c64f, instance id: 019f6666-bb8a-7527-88c8-904e53c1c650 2026-07-15 153136.627Z WARN (starrocks-mysql-nio-pool-14|750) [StmtExecutor.execute():1088] retry 1 times. stmt: select count(*) from helpdesk_tickets 2026-07-15 153136.627Z WARN (starrocks-mysql-nio-pool-14|750) [StmtExecutor.execute():1127] Query 019f6666-bb8a-7527-88c8-904e53c1c64f failed. Planner profile : Planner: - -- Parser[1] 0 - -- Total[1] 6ms - -- Analyzer[1] 0 - -- Lock[1] 0 - -- AnalyzeDatabase[1] 0 - -- AnalyzeTemporaryTable[1] 0 - -- AnalyzeTable[1] 0 - -- Transformer[1] 0 - -- Optimizer[1] 4ms - -- MVPreprocess[1] 0 - -- MVTextRewrite[1] 0 - -- RuleBaseOptimize[1] 2ms - -- CostBaseOptimize[1] 0 - -- PhysicalRewrite[1] 0 - -- DynamicRewrite[1] 0 - -- PlanValidate[1] 0 - -- TypeChecker[1] 0 - -- ConditionalTypeChecker[1] 0 - -- CTEUniqueChecker[1] 0 - -- InputDependenciesChecker[1] 0 - -- ColumnReuseChecker[1] 0 - -- ExecPlanBuild[1] 0 - -- Pending[1] 0 - -- Prepare[1] 0 - -- Deploy[1] 1m - -- DeployLockInternalTime[1] 1m - -- DeploySerializeConcurrencyTime[2] 1ms - -- DeployStageByStageTime[5] 1ms - -- DeployWaitTime[5] 1m - -- DeployAsyncSendTime[7] 1ms - -- Analyzer[1] 0 - -- Lock[1] 0 - -- AnalyzeDatabase[1] 0 - -- AnalyzeTemporaryTable[1] 0 - -- AnalyzeTable[1] 0 - -- Transformer[1] 0 - -- Optimizer[1] 4ms - -- MVPreprocess[1] 0 - -- MVTextRewrite[1] 0 - -- RuleBaseOptimize[1] 2ms - -- CostBaseOptimize[1] 1ms - -- PhysicalRewrite[1] 0 - -- DynamicRewrite[1] 0 - -- PlanValidate[1] 0 - -- TypeChecker[1] 0 - -- ConditionalTypeChecker[1] 0 - -- CTEUniqueChecker[1] 0 - -- InputDependenciesChecker[1] 0 - -- ColumnReuseChecker[1] 0 - -- ExecPlanBuild[1] 0 - DeployDataSize: 43630 Reason: 2026-07-15 153137.840Z WARN (update-blacklist-thread|148) [HostBlacklist.remove():105] remove black list: 10006, at: 2026-07-15T153137.840492543 2026-07-15 153137.840Z WARN (update-blacklist-thread|148) [HostBlacklist.refresh():230] remove nodeID 10006 from blacklist 2026-07-15 153236.717Z WARN (pool-47-thread-1|175) [RpcTimerTask.run():73] correlationId:290 timeout with bound channel =>[id: 0x43c7406e, L/10.99.111.11246018 - R/10.99.111.938060] 2026-07-15 153236.717Z WARN (pool-47-thread-1|175) [RpcTimerTask.run():73] correlationId:291 timeout with bound channel =>[id: 0xf411c7ff, L/10.99.111.11257846 - R/10.99.111.748060] 2026-07-15 153236.717Z WARN (pool-47-thread-1|175) [RpcTimerTask.run():73] correlationId:292 timeout with bound channel =>[id: 0x8558ff51, L/10.99.111.11254394 - R/10.99.111.1118060] 2026-07-15 153236.717Z WARN (pool-47-thread-1|175) [RpcTimerTask.run():73] correlationId:293 timeout with bound channel =>[id: 0x982d629c, L/10.99.111.11242806 - R/10.99.111.1088060] 2026-07-15 153236.717Z WARN (starrocks-mysql-nio-pool-14|750) [FragmentInstanceExecState.waitForDeploymentCompletion():299] catch a execute exception java.util.concurrent.ExecutionException: A error occurred: errorCode=62 errorMessage:method request time out, please check 'onceTalkTimeout' property. current value is:60000(MILLISECONDS) correlationId:290 timeout with bound channel =>[id: 0x43c7406e, L/10.99.111.11246018 - R/10.99.111.938060] at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:578) at com.starrocks.qe.scheduler.dag.FragmentInstanceExecState.waitForDeploymentCompletion(FragmentInstanceExecState.java:291) at com.starrocks.qe.scheduler.Deployer.waitForDeploymentCompletion(Deployer.java:326) at com.starrocks.qe.scheduler.Deployer.deployFragments(Deployer.java:213) at com.starrocks.qe.scheduler.dag.AllAtOnceExecutionSchedule.schedule(AllAtOnceExecutionSchedule.java:134) at com.starrocks.qe.DefaultCoordinator.deliverExecFragments(DefaultCoordinator.java:688) at com.starrocks.qe.DefaultCoordinator.startScheduling(DefaultCoordinator.java:581) at com.starrocks.qe.scheduler.Coordinator.execWithQueryDeployExecutor(Coordinator.java:108) at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1876) at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:1069) at com.starrocks.qe.ConnectProcessor.executeQueryAttempt(ConnectProcessor.java:600) at com.starrocks.qe.ConnectProcessor.runWithParserStageRetry(ConnectProcessor.java:494) at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:431) at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:831) at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1216) at com.starrocks.mysql.nio.MySQLReadListener.handleRequest(MySQLReadListener.java:152) at com.starrocks.mysql.nio.MySQLReadListener.lambda$handleEvent$0(MySQLReadListener.java:92) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: com.baidu.jprotobuf.pbrpc.ErrorDataException: A error occurred: errorCode=62 errorMessage:method request time out, please check 'onceTalkTimeout' property. current value is:60000(MILLISECONDS) correlationId:290 timeout with bound channel =>[id: 0x43c7406e, L/10.99.111.11246018 - R/10.99.111.938060] at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.doWaitCallback(ProtobufRpcProxy.java:651) at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.access$0(ProtobufRpcProxy.java:611) at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:576) ... 19 more 2026-07-15 153236.718Z WARN (starrocks-mysql-nio-pool-14|750) [FragmentInstanceExecState.waitForDeploymentCompletion():325] exec plan fragment failed, errmsg=exec rpc error. backend [id=10022] [host=10.99.111.93], code=THRIFT_RPC_ERROR, fragmentId=F00, backend=10.99.111.93:9060 2026-07-15 153236.718Z WARN (starrocks-mysql-nio-pool-14|750) [FragmentInstanceExecState.waitForDeploymentCompletion():299] catch a execute exception java.util.concurrent.ExecutionException: A error occurred: errorCode=62 errorMessage:method request time out, please check 'onceTalkTimeout' property. current value is:60000(MILLISECONDS) correlationId:291 timeout with bound channel =>[id: 0xf411c7ff, L/10.99.111.11257846 - R/10.99.111.748060] at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:578) at com.starrocks.qe.scheduler.dag.FragmentInstanceExecState.waitForDeploymentCompletion(FragmentInstanceExecState.java:291) at com.starrocks.qe.scheduler.Deployer.waitForDeploymentCompletion(Deployer.java:326) at com.starrocks.qe.scheduler.Deployer.deployFragments(Deployer.java:213) at com.starrocks.qe.scheduler.dag.AllAtOnceExecutionSchedule.schedule(AllAtOnceExecutionSchedule.java:134) at com.starrocks.qe.DefaultCoordinator.deliverExecFragments(DefaultCoordinator.java:688) at com.starrocks.qe.DefaultCoordinator.startScheduling(DefaultCoordinator.java:581) at com.starrocks.qe.scheduler.Coordinator.execWithQueryDeployExecutor(Coordinator.java:108) at com.starrocks.qe.StmtExecutor.handleQueryStmt(StmtExecutor.java:1876) at com.starrocks.qe.StmtExecutor.execute(StmtExecutor.java:1069) at com.starrocks.qe.ConnectProcessor.executeQueryAttempt(ConnectProcessor.java:600) at com.starrocks.qe.ConnectProcessor.runWithParserStageRetry(ConnectProcessor.java:494) at com.starrocks.qe.ConnectProcessor.handleQuery(ConnectProcessor.java:431) at com.starrocks.qe.ConnectProcessor.dispatch(ConnectProcessor.java:831) at com.starrocks.qe.ConnectProcessor.processOnce(ConnectProcessor.java:1216) at com.starrocks.mysql.nio.MySQLReadListener.handleRequest(MySQLReadListener.java:152) at com.starrocks.mysql.nio.MySQLReadListener.lambda$handleEvent$0(MySQLReadListener.java:92) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) Caused by: com.baidu.jprotobuf.pbrpc.ErrorDataException: A error occurred: errorCode=62 errorMessage:method request time out, please check 'onceTalkTimeout' property. current value is:60000(MILLISECONDS) correlationId:291 timeout with bound channel =>[id: 0xf411c7ff, L/10.99.111.11257846 - R/10.99.111.748060] at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.doWaitCallback(ProtobufRpcProxy.java:651) at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.access$0(ProtobufRpcProxy.java:611) at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy$2.get(ProtobufRpcProxy.java:576) ... 19 more
r
The error logs and query profile indicate that your Follower FEs are timing out while trying to "Deploy" (send execution plans) to the Backends (BEs). Specifically, the log 
errorCode=62 errorMessage:method request time out... current value is:60000(MILLISECONDS)
shows that the Follower FE (at 
10.99.111.112
) waited for 60 seconds (the default 
brpc_send_plan_fragment_timeout_ms
) for a response from the BE (at 
10.99.111.93:8060
) but received nothing. Since this query works on the Leader FE but fails on the Followers, the issue is almost certainly network connectivity between the new Follower nodes and the BEs. 1. Verify Network Connectivity (Port 8060) Every FE node (Leader, Follower, or Observer) must be able to reach every BE node on its bRPC port (default 
8060
). Run this test from both Follower FE machines (
.112
and 
.102
):
Copy code
bash
# Test connectivity to a BE that failed in your logs
telnet 10.99.111.93 8060
telnet 10.99.111.108 8060
If this fails to connect immediately, your network/firewall is blocking the Follower FEs. Ensure that the Security Groups or iptables on your BE nodes allow traffic on port 
8060
from the entire FE subnet (or specifically from the two new Follower IPs). 2. Verify Reverse Connectivity (Port 9020) After an FE sends a plan to a BE, the BE often needs to communicate back to the FE's RpcPort (default 
9020
). * Ensure all BE nodes can reach all Follower FEs on port 9020. * In your 
SHOW FRONTENDS
output, the 
RpcPort
for all FEs is 
9020
. Ensure this is open in both directions. 3. Check 
priority_networks
Configuration If your nodes have multiple network interfaces, the FEs might be trying to communicate using an IP address that is not routable to the BEs. * Check 
fe.conf
on the Followers and 
be.conf
on all BEs. * Ensure 
priority_networks
is set to the correct CIDR (e.g., 
10.99.111.0/24
). * If this is not set, the FE might be identifying itself to the BEs with a "local" or "public" IP that the BEs cannot talk back to. 4. Adjust RPC Timeout (Workaround) If your network is inherently slow or congested (though 60s is extremely high for a simple 
COUNT(*)
), you can try increasing the timeout in 
fe.conf
on all FE nodes:
Copy code
conf
# Increase fragment deployment timeout (default is 60000)
brpc_send_plan_fragment_timeout_ms = 120000
Note: This is usually a band-aid. A 60-second timeout for a 43KB plan fragment (as seen in your logs) strongly suggests dropped packets or a firewall "DROP" rule rather than actual slowness. Summary Recommendation The 
DeployWaitTime[5] 1m
in your profile is the smoking gun. It means the FE is hanging while trying to hand off the query to the BE. 1. Primary Cause: Firewall/Network ACL blocking port 8060 from the new Follower IPs to the BEs. 2. Secondary Cause: 
priority_networks
mismatch causing the FE to bind to the wrong interface. Check your infrastructure's network rules first, as the Leader FE's working status implies the BEs themselves are healthy. is healthy but simply unreachable by the new nodes. References • administration/management/FE_configuration.md
v
@Rocky, Follower BE is able connect to BE nodes: [varnitkumarchoukade@ip-10-99-111-112 log]$ nc -zvw3 10.99.111.103 8060 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.103:8060. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. [varnitkumarchoukade@ip-10-99-111-112 log]$ nc -zvw3 10.99.111.93 8060 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.93:8060. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. [varnitkumarchoukade@ip-10-99-111-112 log]$ nc -zvw3 10.99.111.108 8060 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.108:8060. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. [varnitkumarchoukade@ip-10-99-111-112 log]$
r
The log and profile output you shared provide a clear "smoking gun." Although you confirmed that the Follower FE can reach the BEs on port 8060, the query is timing out during the Deployment phase: * 
DeployWaitTime: 1m
: The Follower FE successfully sent the execution plan to the BE (taking only 1ms), but it then waited for the full 60-second timeout (
brpc_send_plan_fragment_timeout_ms
) for the BE to acknowledge or respond. * 
errorCode=62 (method request time out)
: This confirms a bRPC-level timeout. The connection was established ("bound channel"), but the BE never sent a response to the "Execute Fragment" request. This behavior (working on Leader, failing on Followers) is almost always due to asymmetric network connectivity or configuration mismatch on the new nodes. 1. Check BE to FE Connectivity (Port 9020) While the FE initiates the request to the BE (port 8060), the BE must be able to communicate back to the FE's 
rpc_port
(default 9020) to report status and complete the handshake. If the BEs cannot reach the new Follower FEs on 9020, the RPC will hang and eventually timeout. Test this from a BE node:
Copy code
bash
# Run this from one of your BE machines (e.g., 10.99.111.93)
nc -zvw3 10.99.111.112 9020
nc -zvw3 10.99.111.102 9020
If this fails, you must open port 9020 on your Follower FE nodes to allow traffic from the BE subnet. 2. Verify 
priority_networks
 If your Follower FE machines have multiple network interfaces (e.g., a public IP and a private 
10.99...
IP), the FE might be identifying itself to the BEs using the "wrong" IP. * Check 
fe.conf
on the Follower FEs. * Ensure 
priority_networks
is explicitly set to match your internal network:
Copy code
conf
    priority_networks = 10.99.111.0/24
* Important: If you change this, you may need to restart the FE. Since these were recently added, verify that they were added using the correct IP in the 
ALTER SYSTEM ADD FOLLOWER
command. 3. Check for Clock Skew In your 
SHOW FRONTENDS
output, the 
LastHeartbeat
is 
21:57:37
, but your log timestamps are 
15:31:36
. While this might just be a timezone difference (UTC vs local), StarRocks is extremely sensitive to clock drift. * Run 
date
on the Leader FE, Follower FE, and one BE simultaneously. * If the difference is greater than 5 seconds, the RPCs may be rejected or ignored. Align them using 
ntp
or 
chrony
. 4. Check BE INFO Logs To see why the BE is not responding, check the logs on the BE side (e.g., on 
10.99.111.93
) at the time of the failure:
Copy code
bash
grep "019f6666-bb8a-7527-88c8-904e53c1c64f" /path/to/be/storage/log/be.INFO
Look for errors like: * 
fail to send report to FE
* 
RPC connection failed
* 
hostname resolution failed
Summary Checklist 1. Open Port 9020 from BEs to all FEs. 2. Confirm 
priority_networks
 is set in 
fe.conf
on the new nodes. 3. Sync clocks across all nodes (FE and BE) to within 5 seconds. 4. Restart the Follower FEs after making configuration changes. If the network is open and clocks are synced, the BE logs (
<http://be.INFO|be.INFO>
) will be the next place to look for the specific reason the BE is "silent" after receiving the plan. References • administration/management/FE_configuration.md • best_practices/query_tuning/query_profile_operator_metrics.md • faq/Deploy_faq.md • administration/management/BE_configuration.md
v
@Rocky, BE is connected to FE: [varnitkumarchoukade@ip-10-99-111-93 ~]$ nc -zvw3 10.99.111.112 9020 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.112:9020. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. [varnitkumarchoukade@ip-10-99-111-93 ~]$ nc -zvw3 10.99.111.102 9020 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.102:9020. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. [varnitkumarchoukade@ip-10-99-111-93 ~]$ nc -zvw3 10.99.111.118 9020 Ncat: Version 7.93 ( nmap.org/ncat ) Ncat: Connected to 10.99.111.118:9020. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. Seeing following errors on BE: W20260715 141336.347200 140677889324608 input_messenger.cpp:377] Fail to read from Socket{id=1326 fd=318 addr=10.99.111.102330028060} (0x7ff0e553da80): Connection timed out [110] W20260715 141412.119489 140680489170496 fragment_context.cpp:225] [Driver] Canceled, query_id=019f661f-dcc9-727b-971f-3bdbfe929cfe, instance_id=019f661f-dcc9-727b-971f-3bdbfe929cff, reas on=InternalError W20260715 141417.227284 140680184964672 thrift_rpc_helper.cpp:118] rpc failed: Rpc error: FE RPC failure, address=TNetworkAddress(hostname=10.99.111.112, port=9020), reason=THRIFT_EAGAIN (timed out), retry times: 0/1, address=TNetworkAddress(hostname=10.99.111.112, port=9020), timeout_ms=5000 W20260715 152617.467266 140680184964672 thrift_rpc_helper.cpp:118] rpc failed: Rpc error: FE RPC failure, address=TNetworkAddress(hostname=10.99.111.112, port=9020), reason=THRIFT_EAGAIN (timed out), retry times: 0/1, address=TNetworkAddress(hostname=10.99.111.112, port=9020), timeout_ms=5000 W20260715 154216.027208 140677156685376 input_messenger.cpp:377] Fail to read from Socket{id=8589935306 fd=326 addr=10.99.111.112460188060} (0x7ff1c8fae700): Connection timed out [110] W20260715 154640.245120 140677851637312 input_messenger.cpp:242] Close Socket{id=8589935511 fd=271 addr=10.99.111.112411908060} (0x7ff1c7b4ea80) due to unknown message: \FF\F4\FF\FD\06 \FF\F4\FF\FD\06\FF\F4\FF\FD\06\r\n\r\n\FF\F4\FF\FD\06\FF\F4\FF\FD\06 W20260715 154755.889296 140677851637312 input_messenger.cpp:242] Close Socket{id=17179870510 fd=271 addr=10.99.111.112458748060} (0x7ff0e553da80) due to unknown message: \FF\F4\FF\FD\0 6^\r\n\r\n\FF\F4\FF\FD\06\FF\F4\FF\FD\06\FF\F4\FF\FD\06\FF\F4\FF\FD\06
r
The log details you provided are extremely helpful. The combination of the FE's 
DeployWaitTime: 1m
 and the BE's 
Connection timed out [110]
 while reading from the FE's socket is a classic symptom of a Network MTU (Maximum Transmission Unit) mismatch or Network Packet Fragmentation issue. The "Smoking Gun" in your logs: 1. FE Side: The FE starts sending the query plan but hangs for exactly 60 seconds (
DeployWaitTime
). It eventually gives up because the BE never acknowledged receipt of the full plan. 2. BE Side: The BE log says 
Fail to read from Socket... Connection timed out [110]
. This means the BE saw the connection start (which is why your 
nc
test works), but it stopped receiving data in the middle of the stream and the connection eventually timed out at the OS level. 3. BE to FE: The 
THRIFT_EAGAIN
on port 9020 indicates the BE is also failing to send status reports back to the FE, likely for the same reason: small packets work, but larger data packets are being dropped. Why does the Leader work but the Followers fail? This often happens when new nodes (your Followers) are provisioned on different physical hosts, subnets, or VPCs that have a different MTU setting than the rest of the cluster (e.g., one set of nodes uses MTU 9000/Jumbo Frames while the other uses 1500). Recommended Actions: 1. Check and Align MTU Settings Check the MTU on the Leader FE, a Follower FE, and a BE:
Copy code
bash
ip addr | grep mtu
* If the Leader and BEs are at 
9000
but the new Followers are at 
1500
(or vice versa), the larger packets sent by StarRocks will be dropped by the network. * Fix: Ensure all nodes in the cluster use the same MTU (usually 
1500
is safest, though 
9000
is faster if supported everywhere). 2. Confirm 
priority_networks
 If your machines have multiple IPs (e.g., a public IP and a private IP), StarRocks might be trying to communicate over the "wrong" interface which has more restrictive firewall rules or MTU. * In 
fe.conf
on the Followers and 
be.conf
on all BEs, ensure this is set:
Copy code
conf
    priority_networks = 10.99.111.0/24
* If the Leader FE was already working, copy its 
priority_networks
setting exactly to the new Follower nodes. 3. Explaining the "Unknown Message" (Telnet) The error 
due to unknown message: \FF\F4\FF\FD\06
in the BE logs is actually Telnet protocol negotiation sequences. * This is caused by your 
telnet
or 
nc
tests (or a load balancer health check) hitting the bRPC port (8060). * *You can ignore this error*; it is just the BE saying "I don't understand the Telnet commands you are sending me." It is not the cause of your query failures. 4. Check for Firewall State Timeouts If you have a hardware firewall or a cloud security group between the FEs and BEs, it might be killing "idle" long-lived RPC connections. _ However, since 
COUNT(_)
fails immediately, the MTU mismatch (Action #1) remains the most likely culprit. Summary Checklist: 1. Compare 
ip addr
MTU values across all nodes. 2. Set 
priority_networks
in 
fe.conf
on the new nodes. 3. Check the BE's 
<http://be.INFO|be.INFO>
log for any "Full GC" or "OOM" messages, though your current logs strongly point to the network layer. Question: Are the new Follower FEs running in the same subnet/VPC as the Leader and the BEs? If not, there is likely a router in between that is dropping fragmented packets. References • administration/management/FE_configuration.md • administration/Meta_recovery.md • faq/Deploy_faq.md
v
@Rocky, Leader:
Copy code
[varnitkumarchoukade@ip-10-99-111-118 ~]$ ip addr | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: enp39s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
Follower:
Copy code
[varnitkumarchoukade@ip-10-99-111-112 log]$ ip addr | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: enp39s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
BE-1:
Copy code
[varnitkumarchoukade@ip-10-99-111-103 ~]$ ip addr | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: enp39s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
BE-2:
Copy code
[varnitkumarchoukade@ip-10-99-111-93 log]$ ip addr | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
2: enp39s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9001 qdisc mq state UP group default qlen 1000
Priority network is same for conf: priority_networks = 10.99.111.0/24 no full GC or OOM in be.INFO: [varnitkumarchoukade@ip-10-99-111-93 log]$ grep "Full GC" be.INFO | tail -20 [varnitkumarchoukade@ip-10-99-111-93 log]$ grep "Full GC" be.INFO [varnitkumarchoukade@ip-10-99-111-93 log]$ grep "OOM" be.INFO [varnitkumarchoukade@ip-10-99-111-93 log]$
r
Since all nodes report an MTU of 
9001
, but your BEs are timing out while reading from the Follower FEs (
Connection timed out [110]
), this strongly indicates a Path MTU Discovery (PMTUD) failure. Even if your nodes are configured for 
9001
, the network path between the new Follower nodes and the BEs might have a segment (like a router, VPC peering, or a security group rule) that doesn't support jumbo frames or is dropping the ICMP "Fragmentation Needed" packets. This allows small packets (like 
nc
or 
ping
) to pass, but larger packets (like execution plans or query results) to be dropped. 1. Perform a Large Payload Ping Test This test forces the network to handle a large packet and will fail if the path doesn't actually support MTU 9001. Run this from a Follower FE (
.112
) to a BE (
.93
):
Copy code
bash
# -M do prevents fragmentation
# -s 8973 is the payload size for a 9001 MTU (9001 - 28 bytes header)
ping -M do -s 8973 10.99.111.93
* If it hangs or fails: The network path between these nodes does not support MTU 9001. You must lower the MTU to 
1500
on all nodes or fix the network infra. * If it works: The issue is at the application/RPC layer. 2. Verify FE Thread Pool & RpcPort The BE logs show 
THRIFT_EAGAIN (timed out)
when calling the Follower FE on port 
9020
. This means the BE reached the port, but the FE didn't respond in time. * Check FE Heap/GC: Even if there's no "Full GC" in the BE, check the 
fe.gc.log
 on the Followers. If the Follower FE is under heavy GC, it will stop responding to Thrift RPCs from the BE. * Check FE configuration: Ensure the 
rpc_port
in 
fe.conf
is explicitly set to 
9020
on the followers. * Check FE thread count: If the Follower is receiving many requests, it might have exhausted its Thrift threads. Look for 
thrift-server-pool
errors in 
fe.warn
. 3. Check for Version Mismatch Run this query on the Leader and check the output for all FEs:
Copy code
sql
SHOW FRONTENDS;
Ensure the 
Version
column is identical for the Leader and both Followers. If you recently added Followers with a different StarRocks version than the Leader, the RPC serialization will fail silently or hang. 4. BE-side "Report" failure In your BE logs, grep for the query ID and look for "report":
Copy code
bash
grep "019f6666-bb8a-7527-88c8-904e53c1c64f" <http://be.INFO|be.INFO> | grep "report"
If you see 
fail to send report to FE
, it confirms the BE is trying to finish the query but cannot reach the FE's internal RPC handler. Summary 1. Lower MTU to 1500 on one Follower FE and one BE as a test. If the query starts working, MTU was the cause. 2. Verify version parity between all FEs. 3. Check 
fe.gc.log
 on the followers for pauses. The fact that 
nc
works but the actual RPC times out is the classic symptom of the network dropping larger data packets while allowing the TCP handshakes (which are small) succeed. References • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/monitoring/metrics.md • quick_start/shared-data.md • faq/Others.md • faq/Sql_faq.md • loading/loading_introduction/troubleshooting_loading.md
3 Views
Open in Slack
PreviousNext
Powered by