@Rocky when powerbi gets data from starrocks via arrow flight service the data is buffered off heap on fe pod? or it is consumed from cn nodes?

When you use Power BI to get data from StarRocks via the Arrow Flight SQL service, the data flow depends on whether the Arrow Flight Proxy is enabled (which is the default behavior). 1. Proxy Mode (Default:

arrow_flight_proxy_enabled = true

) In this mode, Power BI connects to the FE (Front End), and the FE acts as a proxy between the client and the CN/BE (Compute Nodes/Backend). * Buffering on FE: The data is indeed buffered on the FE pod. Because StarRocks uses the Apache Arrow library for this service, it utilizes off-heap memory (Direct Memory) for data transfer. * JVM Configuration: This is why you must add

--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED

to the FE's

JAVA_OPTS

. This flag allows the Arrow library on the FE to manage off-heap memory buffers efficiently. * Data Path: CN/BE → FE (Off-heap buffer) → Power BI. 2. Direct Mode (

arrow_flight_proxy_enabled = false

) If you disable the proxy, the FE only handles the initial handshake and then provides "tickets" that point the client directly to the CN/BE nodes. * Consumption from CN: In this case, Power BI consumes data directly from the CN nodes. * Network Requirement: This requires that the Power BI environment has direct network access to the IP addresses of all CN/BE nodes, which is often not the case in Kubernetes or restricted VPC environments. * Data Path: CN/BE → Power BI. Summary for Power BI Since Power BI usually connects to a single endpoint (like a Load Balancer pointing to StarRocks FEs), Proxy Mode is the most common configuration. In this setup: * Yes, the data is buffered in off-heap memory on the FE pod during transit. * This can lead to increased memory pressure on the FE if you are pulling very large datasets simultaneously, so ensure your FE pods have sufficient memory overhead beyond their heap size. How to check/set your mode:

sql
-- Check current setting
SHOW VARIABLES LIKE 'arrow_flight_proxy_enabled';

-- Disable proxy (to read directly from CNs, if network allows)
SET GLOBAL arrow_flight_proxy_enabled = false;

References • administration/management/BE_configuration.md • unloading/arrow_flight.md • administration/management/FE_configuration.md

@Rocky so whats the use of arrow fligh proxy enabled?

The primary use of

arrow_flight_proxy_enabled

is to bridge network connectivity gaps between your client (like Power BI) and the StarRocks compute nodes (CN/BE). Here is the specific breakdown of why you would use it: 1. Solving "Hidden" Node Issues (The Main Use Case) In many environments (especially Kubernetes or restricted VPCs), only the StarRocks FE (Front End) is exposed via a Load Balancer. The CN/BE nodes are often on a private internal network that the Power BI machine cannot reach directly. * With Proxy (Enabled): Power BI only needs to talk to the FE. The FE fetches the data from the CNs and hands it to Power BI. * Without Proxy (Disabled): The FE tells Power BI: "Go get this data from CN Node 10.0.0.5." If Power BI cannot "ping"

10.0.0.5

, the connection will fail. 2. Single Endpoint Management Managing firewall rules or SSL certificates for 100+ individual CN nodes is difficult. By enabling the proxy, you only need to manage the connection to the FE/Load Balancer endpoint. 3. The Performance vs. Resource Trade-off Choosing whether to enable it is a balance between performance and architectural simplicity: Feature | Proxy Enabled (

true

) | Proxy Disabled (

false

) Network Path | Client → FE → CN | Client → CN (Direct) Connectivity | Client only needs FE access | Client needs access to all CN IPs Throughput | ~8-10% slower (due to extra hop) | Maximum performance (zero-copy) FE Impact | High: Buffers data in off-heap memory | Low: FE only handles metadata Best For | Kubernetes, Power BI via Gateway | High-perf Data Science, Spark Summary You keep it enabled (default) if your Power BI environment cannot directly reach every single StarRocks compute node IP address. You only disable it if you have a flat network where every machine can talk to every other machine and you want the absolute highest data transfer speeds possible. Note: If you use it in Proxy mode, remember to monitor your FE's non-heap memory usage, as that is where the Arrow data is buffered during the transfer. References • unloading/arrow_flight.md

@Rocky odbc vs adbc why odbc does not eat off heap mmemory and arrow flight does?

The difference in memory behavior between ODBC (MySQL protocol) and Arrow Flight (ADBC) comes down to how the data is formatted and which library manages the memory. 1. Arrow Flight (ADBC): High-Performance Off-Heap Arrow Flight is designed for zero-copy data transfer. To achieve this, it uses the Apache Arrow memory format. * Off-Heap by Design: The Apache Arrow library manages its own memory outside of the standard JVM heap (using "Direct Memory"). This allows the FE to send data directly to the network socket without the Java Garbage Collector (GC) interfering or moving data around. * Why it "eats" memory: When

arrow_flight_proxy_enabled

is on, the FE acts as a middleman. It receives Arrow buffers from the CN/BE and holds them in this off-heap area before passing them to Power BI. * The JVM Flag: This is exactly why you need the

--add-opens=java.base/java.nio=...

flag. It gives the Arrow library permission to access the low-level Java APIs required to manage that off-heap memory. 2. ODBC (MySQL Protocol): Heap-Based Serialization Traditional ODBC connections to StarRocks use the MySQL wire protocol. * Row-Based Serialization: In this mode, StarRocks must convert its internal columnar data into a row-based format. This serialization usually happens within the standard memory limits of the process. * Standard Buffering: While the underlying network layer (like Netty) uses some off-heap memory for socket buffers, the actual "data processing" and "result set construction" for the MySQL protocol typically happen in the JVM Heap or via standard temporary buffers that are quickly released. * No Arrow Overhead: Since it doesn't use the Apache Arrow library to store the result set, it doesn't utilize the specialized Arrow off-heap memory pool that requires specific configuration and monitoring. Summary Comparison Feature | ODBC / MySQL Protocol | Arrow Flight / ADBC Data Format | Row-based (Serialized) | Columnar (Arrow) Memory Location | Mostly JVM Heap / Socket Buffers | Off-Heap (Direct Memory) Management | JVM Garbage Collector | Apache Arrow Memory Manager Why? | Legacy compatibility, slower. | Speed. Zero-copy requires direct memory access. FE Risk | High CPU (due to serialization). | High Memory (due to off-heap buffering). In short: Arrow Flight "eats" off-heap memory because that's the "secret sauce" that makes it 10x-100x faster than ODBC. It trades memory overhead for extreme data transfer speed by bypassing the JVM's heap limitations. References • unloading/arrow_flight.md

@Rocky so odbc stays within jvm and onheap and is managed and arrow flight by default is not managed and is accumulating offheap and via java options it can be handled?

The comparison between ODBC (MySQL protocol) and Arrow Flight (ADBC) essentially describes two different memory management philosophies within the same FE process. 1. ODBC (MySQL Protocol): Primarily Managed On-Heap When you use ODBC, the FE communicates using the MySQL wire protocol. * On-Heap Focus: The FE must transform columnar data into row-based bytes. This logic and the resulting objects are stored in the JVM Heap. * GC Managed: Because this data lives in the heap, the Java Garbage Collector (GC) is responsible for cleaning it up. If you pull too much data, you see "High Heap Usage" or "Full GC" in your FE logs. * Off-Heap usage: While it uses some off-heap memory (via Netty for network sockets), this is usually a small, fixed-size buffer that doesn't scale linearly with the dataset size. 2. Arrow Flight (ADBC): Primarily Managed Off-Heap Arrow Flight is designed to bypass the limitations of the JVM heap to achieve higher speeds. * Off-Heap Focus: The data is stored in Direct Memory (Off-Heap) using the Apache Arrow format. It does not sit in the JVM heap. * Library Managed: It is not managed by the JVM Garbage Collector. Instead, it is managed by the Apache Arrow Memory Manager. This avoids the "Stop-the-World" GC pauses that happen when the heap gets full. * "Accumulation": It doesn't "accumulate" forever (like a memory leak). It buffers the data only while the query is active and the client (Power BI) is consuming it. Once the data is sent, the Arrow memory manager releases the buffer back to the OS. 3. How to "Handle" it via Java Options You are correct that Arrow Flight requires specific configuration because it operates outside the standard heap "sandbox": *

--add-opens...

: This flag is the "permission slip." It allows the Arrow library to reach into the internal Java APIs (like

java.nio

) to manipulate off-heap memory blocks directly. Without this, Arrow Flight will fail with an error. *

-XX:MaxDirectMemorySize

: This is the "throttle." While the GC doesn't manage this memory, the JVM still tracks it. You can set this in

fe.conf

(under

JAVA_OPTS

) to cap how much off-heap memory the Arrow service is allowed to use. _ _Example:*

JAVA_OPTS="-Xmx16G -XX:MaxDirectMemorySize=8G ..."

*

MALLOC_ARENA_MAX=1

: Often recommended in

fe.conf

to prevent the underlying system's

glibc

from over-allocating memory arenas, which can lead to the FE process being killed by the OS (OOM Killer) even if the heap is fine. Summary * ODBC is "managed" by the Garbage Collector and constrained by Heap size (

-Xmx

). * Arrow Flight is "managed" by the Arrow Library and constrained by Direct Memory size (

-XX:MaxDirectMemorySize

). If you are using Power BI with the FE as a proxy, you should ensure your FE pod has enough total RAM to accommodate both your Heap (metadata/catalog) and your Direct Memory (Arrow Flight buffers). References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/resource_management/Memory_management.md • faq/Others.md • administration/management/FE_configuration.md

@Rocky

MaxDirectMemorySize

so this limit wont eat all the pods ram and arrow flight will consume data in chunks

Yes, you have identified the core mechanism correctly. Here is how it works to protect your StarRocks FE from crashing while maintaining high performance. 1. The Safety Cap:

MaxDirectMemorySize

The

MaxDirectMemorySize

is a hard limit for the JVM's off-heap memory. * The Guardrail: If you set

-XX:MaxDirectMemorySize=4G

in your

fe.conf

, the Arrow Flight service is physically unable to claim more than 4GB of RAM from the host, regardless of how large the query result is. * Preventing Pod Crashes: Without this limit, the Arrow library might keep requesting memory until the Kubernetes OOM-Killer terminates the entire FE pod. With the limit, the query will simply fail with an

OutOfMemoryError: Direct buffer memory

, but the FE remains alive to serve other users. 2. The Streaming Mechanism: "Data in Chunks" Arrow Flight does not wait to download the entire result set before sending it to you. It uses a Streaming Pipeline: * BE to FE (Proxy): The Compute Nodes (BE) send data to the FE in RecordBatches (chunks of several thousand rows). * FE to Client: As soon as the FE receives a batch, it immediately "proxies" (pushes) it to the client (like Power BI). * Memory Recycling: Once a chunk is successfully sent over the network to the client, the Arrow Memory Manager frees that off-heap memory so it can be reused for the next chunk. 3. Why this matters for your Pod Because of this "sliding window" of chunks, the FE memory usage for Arrow Flight depends on concurrency, not total data size: * Scenario A: You pull 100GB of data with 1 user. The FE might only use 256MB of off-heap memory at any given time because it is streaming small chunks and recycling the memory immediately. * Scenario B: You have 50 users all pulling 1GB at the same time. The FE usage will spike because it is holding the "active" chunks for 50 different streams simultaneously. Best Practice Summary To ensure your FE pod is stable: 1. Set the limit: Always define

-XX:MaxDirectMemorySize

in your

JAVA_OPTS

. A common rule of thumb is to give 25-30% of your Pod's total RAM to Direct Memory and the rest to the JVM Heap (

-Xmx

). 2. Monitor: Watch the FE's non-heap memory metrics. If you see frequent

Direct buffer memory

errors, you need to either increase the pod's RAM or reduce the number of concurrent large exports. 3. OS Settings: Ensure

MALLOC_ARENA_MAX=1

is set in your environment to prevent the underlying Linux memory allocator from fragmenting the off-heap space. References • unloading/arrow_flight.md • administration/management/resource_management/Memory_management.md • faq/fe_mem_faq.md References • unloading/arrow_flight.md

@Rocky now fe does not restart and OOM is fixed but I have error "026-08-12 200942.598+05:00 WARN (grpc-nio-worker-ELG-6-6|522) [DefaultPromise.notifyListener0():607] An exception was thrown by io.grpc.netty.NettyServerStream$Sink$$Lambda$4160/0x000071fe08632700.operationComplete() java.lang.NoSuchMethodError: 'boolean io.grpc.netty.NettyServerStream$TransportState.isStreamDeallocated()' at io.grpc.netty.NettyServerStream$TransportState.handleWriteFutureFailures(NettyServerStream.java:209) at io.grpc.netty.NettyServerStream$TransportState.access$400(NettyServerStream.java:139) at io.grpc.netty.NettyServerStream$Sink.lambda$writeTrailers$2(NettyServerStream.java:126) at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:604) at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:571) at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:506) at io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:650) at io.netty.util.concurrent.DefaultPromise.setFailure0(DefaultPromise.java:643) at io.netty.util.concurrent.DefaultPromise.tryFailure(DefaultPromise.java:132) at io.netty.channel.AbstractChannel$AbstractUnsafe.safeSetFailure(AbstractChannel.java:997) at io.netty.channel.AbstractChannel$AbstractUnsafe.write(AbstractChannel.java:858) at io.netty.channel.DefaultChannelPipeline$HeadContext.write(DefaultChannelPipeline.java:1314) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:889) at io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:875) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:984) at io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:868) at io.netty.channel.DefaultChannelPipeline.write(DefaultChannelPipeline.java:964) at io.netty.channel.AbstractChannel.write(AbstractChannel.java:300) at io.grpc.netty.WriteQueue$AbstractQueuedCommand.run(WriteQueue.java:206) at io.grpc.netty.WriteQueue.flush(WriteQueue.java:128) at io.grpc.netty.WriteQueue.access$000(WriteQueue.java:35) at io.grpc.netty.WriteQueue$1.run(WriteQueue.java:47) at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:472) at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:566) at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998) at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.base/java.lang.Thread.run(Thread.java:840) 2026-08-12 200942.633+05:00 WARN (arrow-flight-executor-1|879) [DefaultCoordinator.getNext():950] get next fail, need cancel. status errorCode CANCELLED UserCancel, query id: 275fdf4f-965f-11f1-8ddc-ea6ee87c39b9 2026-08-12 200942.634+05:00 WARN (arrow-flight-executor-1|879) [DefaultCoordinator.dealStatusToTryRetry():913] query 275fdf4f-965f-11f1-8ddc-ea6ee87c39b9 failed: Close Arrow Flight SQL session via RPC request"

The error

java.lang.NoSuchMethodError: 'boolean io.grpc.netty.NettyServerStream$TransportState.isStreamDeallocated()'

is a known issue (StarRocks GitHub Issue #74577) caused by a dependency version mismatch within the StarRocks FE. Why this is happening: 1. Version Conflict: Recent versions of StarRocks upgraded the Arrow Flight (flight-core) library to version 18.0.0. This library pulls in

grpc-netty

version 1.65.0. 2. The Missing Method: The method

isStreamDeallocated()

was introduced in gRPC 1.65.0. 3. JAR Hell: The StarRocks FE classpath still contains an older version of

grpc-core

(likely 1.63.0 or earlier) that does not yet have this method. Because the "core" and "netty" parts of gRPC must have matching versions, the runtime fails when it tries to call that specific cleanup method. 4. The Result: When the query finishes or is cancelled, Arrow Flight tries to clean up the network stream. The method call fails, causing the

NoSuchMethodError

and preventing the session from closing gracefully, which is why you see the

CANCELLED UserCancel

warnings immediately after. How to fix it: This is a bug in the StarRocks distribution package where incompatible JAR files were bundled together. * Short-term / Workaround: _ If you have control over the FE

lib/

directory, you can check if there are multiple versions of

grpc-core-_.jar

and

grpc-netty-*.jar

. However, manually replacing JARs in the StarRocks installation is risky and not recommended for production. * Recommended Solution: * Upgrade StarRocks: This issue has been identified and is being fixed in recent patch releases (e.g., check for the latest versions of 3.3.x, 3.4.x, or 3.5.x). * Check your StarRocks version. If you are on a version released around May/June 2026, you are likely hitting this specific bug. * Configuration Check: * Ensure your

fe.conf

has the necessary

--add-opens

flags (as discussed earlier) to ensure the memory management doesn't trigger secondary issues, though this won't fix the

NoSuchMethodError

. If you are on the latest available patch and still see this, it is recommended to report your specific StarRocks version in the community Slack or GitHub issue #74577 to verify if the fix has been backported to your branch. References • unloading/arrow_flight.md • faq/Deploy_faq.md • administration/management/FE_configuration.md References • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • faq/Deploy_faq.md • developers/trace-tools/Trace.md • faq/resource_isolation_faq.md • deployment/environment_configurations.md

@Rocky I'm on version 3.5.16

The fact that your FE is no longer restarting and the OOM is gone confirms that the Off-Heap memory management (

MaxDirectMemorySize

and

--add-opens

) is working correctly. You have successfully moved the data transfer load out of the JVM Heap. However, the new error you are seeing—`java.lang.NoSuchMethodError: 'boolean io.grpc.netty.NettyServerStream$TransportState.isStreamDeallocated()'`—is a packaging bug in the StarRocks 3.5.x distribution (specifically appearing in recent builds like 3.5.16). Why this is happening This is a "JAR Hell" conflict. The Arrow Flight service in StarRocks 3.5 uses gRPC 1.65.x (or newer), which introduced the

isStreamDeallocated()

method. However, the StarRocks FE classpath still contains an older version of a dependent gRPC or Netty JAR that lacks this method. When the query finishes or is cancelled (the

CANCELLED UserCancel

in your logs), the system tries to clean up the network stream. It calls this method, fails to find it, and throws the error. How to handle this in 3.5.16 Since you are on a very specific recent version (3.5.16), here are the recommended steps: 1. Check for Duplicate JARs: Navigate to your FE installation directory:

fe/lib/

. Look for multiple versions of gRPC or Netty JARs. * Example:

grpc-netty-1.65.x.jar

vs

grpc-core-1.5x.x.jar

. * If you see two different versions of the same library (e.g., two versions of

grpc-netty

), this is the source of the conflict. 2. It is a "Clean-up" Error: Notice that the error occurs during

handleWriteFutureFailures

and

operationComplete

. This means the query data has likely already been sent or the user has already clicked "Cancel" in Power BI. The error is preventing a clean "handshake" to close the session, but it usually doesn't affect the data that was already successfully transferred. 3. Check for 3.5.17+ Patch: This issue was reported in the StarRocks community as a regression in the 3.5 branch. It is typically fixed by aligning all gRPC dependencies to the same version (1.65.0+). Check if a 3.5.17 or a hotfix for 3.5.16 is available in your repository. 4. Workaround (Configuration): Ensure your

fe.conf

includes the specific

add-opens

required for the latest Arrow versions. While it won't fix the missing method, it ensures the gRPC/Arrow interaction doesn't fail for other reasons:

bash
    JAVA_OPTS="--add-opens=java.base/java.nio=org.apache.arrow.memory.core,ALL-UNNAMED"

Summary of your status: * Memory: Fixed. The streaming is happening in chunks off-heap. * Stability: Improved. The FE is not crashing/restarting. * Current Error: Packaging Bug. A library conflict in version 3.5.16 is causing failures during connection cleanup. This needs a patch release from the StarRocks side or a manual cleanup of the

fe/lib

directory. References • unloading/arrow_flight.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md References • unloading/arrow_flight.md

@Rocky may be there is some timeout?

The

UserCancel

and

Close Arrow Flight SQL session via RPC request

logs, combined with your question about timeouts, strongly suggest that while you've fixed the OOM, the query is now being terminated prematurely by a timeout—either on the server side, the client side, or a network intermediary. The

java.lang.NoSuchMethodError

you see is a *secondary failure*; it happens when StarRocks tries to clean up the connection after it has already been cancelled. 1. The Most Likely Culprit:

query_timeout

StarRocks has a default session timeout for all queries. * Default: 300 seconds (5 minutes). * Behavior: If your large data transfer takes longer than 5 minutes to stream the "chunks" to the client, StarRocks will kill the query. * Log Clue: If your query consistently fails around the 5-minute mark, this is the cause. * Fix: Increase the timeout for your session or globally:

sql
    -- For the current session (recommended for testing)
    SET query_timeout = 3600; -- 1 hour

    -- To check current value
    SHOW VARIABLES LIKE 'query_timeout';

2. Infrastructure/Load Balancer Timeouts Since you mentioned "Pods," you are likely running in Kubernetes. * Idle Timeout: If you are accessing StarRocks through a Kubernetes Ingress (like Nginx) or a Cloud Load Balancer (AWS ELB/NLB), they often have default idle timeouts (usually 60 seconds). * The Conflict: Arrow Flight streams data. If the client (like Power BI) is slow to process a chunk, the connection might appear "idle" to a Load Balancer, causing it to sever the TCP connection. * Result: When the connection drops, the StarRocks FE receives a "connection reset," which it logs as a

UserCancel

because the RPC stream was closed from the other end. 3. Client-Side Timeouts If you are using the Arrow Flight SQL JDBC driver: * Check if your BI tool (Power BI, Tableau, etc.) has its own command timeout settings. * In the JDBC connection string, you can sometimes specify timeouts, though

query_timeout

at the SQL level is usually more effective. 4. Why the

NoSuchMethodError

appears This error is a confirmed bug in the StarRocks 3.5.16 distribution where some internal libraries (gRPC) have version mismatches. * Normal flow: Query finishes -> FE cleans up gRPC stream -> Success. * Your flow: Query hits a timeout -> FE receives cancel signal -> FE tries to clean up the stream -> Cleanup fails due to the bug (

NoSuchMethodError

). The

NoSuchMethodError

is not the cause of your query failing; it's just preventing a "clean" exit after the timeout occurs. Summary & Next Steps 1. Time the failure: Use a stopwatch. If it's exactly 300s, it's

query_timeout

. If it's exactly 60s, it's likely your K8s Ingress/Load Balancer. 2. Increase SR Timeout: Run

SET query_timeout = 3600;

before your export. 3. Check Ingress: If using Nginx Ingress, add annotations to increase

proxy-read-timeout

and

proxy-send-timeout

. References • unloading/arrow_flight.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md References • unloading/Flink_connector.md • unloading/arrow_flight.md

@Rocky but query does not finish it stops working after 50%

If the query consistently stops around 50%, it is likely hitting a resource limit or a network timeout rather than a simple OOM crash. Since you've fixed the FE OOM, the process is staying alive, but the query itself is being terminated. Here are the most likely reasons why an Arrow Flight query "stalls" or cancels halfway: 1. The 5-Minute

query_timeout

This is the most common cause. StarRocks has a default limit of 300 seconds (5 minutes) for any single query. * The Symptom: If your data transfer is large, it might reach 50% exactly when the 5-minute mark hits. * The Fix: Increase the timeout for the session you are using for the transfer:

sql
    SET query_timeout = 3600; -- Set to 1 hour

Note: If you are using a BI tool like Power BI, you may need to set this as a "Session SQL" or "Connection Init SQL" if the tool supports it. 2. BE-Side

query_mem_limit

Even though you fixed the FE OOM, each query also has a memory limit on the Backends (BE). * The Symptom: If the query needs to buffer a lot of data before sending it via Arrow Flight, it might hit the

query_mem_limit

on the BE. * The Fix: Try increasing the memory limit for the query:

sql
    SET query_mem_limit = 8589934592; -- 8GB (adjust based on your BE RAM)

3. Load Balancer / Idle Timeouts If you are running in Kubernetes or behind an AWS/Azure Load Balancer, there is often an idle timeout (commonly 60 seconds). * The Symptom: Arrow Flight sends data in "batches." If the BE is busy preparing the next batch and the connection remains "quiet" for too long, the Load Balancer will kill the connection. * The Result: The StarRocks FE sees the connection drop and logs it as

CANCELLED UserCancel

(because the "user" side of the socket disappeared). * The Fix: * If using Nginx Ingress, increase

proxy-read-timeout

and

proxy-send-timeout

. * If using a Cloud Load Balancer, increase the "Idle Timeout" setting to 3600s. 4. Arrow Flight Proxy Bottleneck By default, StarRocks uses Proxy Mode (

arrow_flight_proxy_enabled=true

), where data flows

BE -> FE -> Client

. This adds overhead and an extra point of failure. * The Fix: If your client (e.g., your laptop or a VM) can "see" the BE nodes directly on the network, try disabling the proxy to let data flow directly from the BEs:

sql
    SET GLOBAL arrow_flight_proxy_enabled = false;

(Note: This requires a new session/connection to take effect). Why you still see the

NoSuchMethodError

As mentioned before, the

NoSuchMethodError

in your logs is not the cause of the query stopping. It is a cleanup bug. 1. The query stops (due to timeout/network/limit). 2. StarRocks tries to close the connection gracefully. 3. The "Close" operation triggers the

NoSuchMethodError

because of a library conflict in version 3.5.16. Recommendation: Start by running

SET query_timeout = 3600;

and check if the query progresses past the 50% mark. If it still stops, check your Load Balancer/Ingress logs for "Connection Reset" or "Timeout" errors. References • unloading/arrow_flight.md • sql-reference/System_variable.md • faq/operation_maintenance_faq.md " errors. References • unloading/arrow_flight.md • sql-reference/System_variable.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • faq/resource_isolation_faq.md • faq/Deploy_faq.md