@Rocky. For some large queries that fetches 23 million rows. Our starrocks runs on 4.0 and deployed in k8s with fe as proxy and many times we get this error: failed to proxy result from BE: failed to read message error or server sendMessage failed with error cancelled doget endpoint etc.

The error messages you're encountering—*"failed to proxy result from BE: failed to read message error"* and *"server sendMessage failed with error cancelled doget endpoint"*—indicate an issue with the FrontEnd (FE) result proxying mechanism, specifically within the Arrow Flight SQL or FE Result Proxy layer. In a Kubernetes environment, StarRocks often uses the FE as a proxy to route data from BEs to clients because BE pods are typically not directly accessible from outside the cluster. For a query fetching 23 million rows, the long-running data stream is likely hitting a timeout or resource limit. Potential Causes and Solutions 1. RPC and Buffer Timeouts Since the error mentions

doget

(an Arrow Flight RPC method) and

sendMessage

, it's likely the internal communication between FE and BE is timing out or being cancelled. * Increase

thrift_rpc_timeout_ms

: This FE configuration controls the default network/socket timeout for internal RPC calls. * Action: In

fe.conf

, increase

thrift_rpc_timeout_ms

(default is 10000ms/10s). Try setting it to

60000

or higher. * Check

brpc_socket_max_unwritten_bytes

: If the BE is producing data faster than the FE can proxy it, the internal buffer may overflow, causing the RPC to fail with a "server overcrowded" or "cancelled" error. * Action: In

be.conf

(or

cn.conf

), increase

brpc_socket_max_unwritten_bytes

. The default is often 1GB; try increasing it to

4294967296

(4GB). 2. K8s Infrastructure Timeouts If your client is connecting to the FE through a Kubernetes Ingress or LoadBalancer, the infrastructure itself may be closing the connection during the long transfer of 23M rows. * Action: Check your Ingress controller (e.g., Nginx) settings for

proxy-read-timeout

and

proxy-send-timeout

. For large data transfers, these often need to be increased to several minutes. 3. Arrow Flight Proxy Configuration If you are using Arrow Flight (indicated by the

doget

error), ensure the proxy is configured correctly to handle high volumes. * Action: Check the session variable

arrow_flight_proxy_enabled

. If it's enabled, the FE handles the data stream. * Memory: Large result sets via proxying can put pressure on FE memory. Monitor the FE JVM heap usage during these queries. If it spikes, you may need to increase the FE memory limit in K8s. 4. Session-Level Timeouts Even if the query hasn't reached the global

query_timeout

, individual packets or parts of the stream might be timing out. * Action: Ensure

query_timeout

is set high enough for 23M rows (e.g.,

SET query_timeout = 3600;

). * Action: In StarRocks 4.0, try adjusting the session variable

mysql_send_packet_timeout_ms

(if using MySQL protocol) or verify the client-side timeout in your Arrow Flight/ADBC driver. Recommended Next Steps 1. Test direct BE access (if possible): If you can run the query from within the K8s cluster and connect directly to a BE (bypassing the FE proxy), check if the error persists. This will isolate whether the issue is with the proxying logic or the query itself. 2. Review FE/BE Logs: Look for Out of Memory (OOM) or Broken Pipe errors in

fe.warn.log

and

<http://be.INFO|be.INFO>

at the exact time of the query failure. 3. Pagination/Export: For results as large as 23M rows, consider using

SELECT ... INTO OUTFILE

or paginating the query, as streaming such a large volume through a single FE proxy can be unstable depending on your hardware be a bottleneck. If the issue persists, check if there are specific fixes in the StarRocks 4.0 patch notes (e.g., 4.0.12) related to Arrow Flight proxying, as this feature has seen significant refinement in recent releases. References • knowledge/usage/best-practice.md • faq/loading/Loading_faq.md • faq/Sql_faq.md