@Rocky I'm benchmarking Arrow Flight SQL from Python against a StarRocks 3.5 cluster, and I seem to have confirmed through both measurement and source code inspection that flight result streams are sent uncompressed. I measured using

psutil

to snapshot NIC bytes before/after `DoGet`: • Arrow table in memory:129 MB (100K rows, XML-heavy column) • Bytes received on NIC:141 MB(the 9% overhead could be gRPC/HTTP2 framing?) • Compression ratio:0.91x, so, no compression applied We tried both

grpc-encoding: gzip

per-call header and

grpc.default_compression_algorithm = 2

as a channel option. Neither changed the wire size. Looking at:

be/src/service/service_be/arrow_flight_sql_service.cpp

the

DoGetStatement

returns:

return std::make_unique<arrow::flight::RecordBatchStream>(reader);

The second parameter (

IpcWriteOptions

) is omitted, so Arrow defaults to

codec = nullptr

(no compression). FE-side

FlightServer.builder

also has no

.compressor()

set, so gRPC-level compression is not negotiated either. Questions 1. Is Arrow Flight compression on the roadmap? 2. Is there a server-side config we've missed that enables it? 3. If not currently supported, is a feature request the right path? The fix appears to be "just" passing`IpcWriteOptions`with a codec to`RecordBatchStream`.

Based on your code investigation and current StarRocks documentation, your observations are correct. Compression for Arrow Flight SQL result streams is not currently implemented in StarRocks 3.5. Here are the specific answers to your questions: 1. Is Arrow Flight compression on the roadmap? There is no publicly documented timeline or specific roadmap item for adding Arrow IPC compression to the Flight SQL interface as of StarRocks 3.5. While the core protocol support was introduced recently (starting in v3.3/v3.4) to optimize large-scale data transfers, the focus has been on the "zero-copy" columnar pipeline rather than payload compression. 2. Is there a server-side config to enable it? No. There are currently no BE or FE configuration parameters (e.g.,

be.conf

,

fe.conf

) or session variables that enable compression for Arrow Flight SQL. * gRPC-level compression: As you noted, setting

grpc-encoding

at the client/channel level does not work because the StarRocks implementation does not currently negotiate or utilize gRPC-level compressors for the Flight service. * IPC-level compression: The code in

arrow_flight_sql_service.cpp

indeed initializes the

RecordBatchStream

using default

IpcWriteOptions

, which defaults to uncompressed output. 3. Is a feature request the right path? Yes. Creating a feature request on the StarRocks GitHub repository is the correct path. Your technical assessment is spot-on: the implementation change involves passing an

IpcWriteOptions

object with a specified

compression

codec (like

LZ4_FRAME

or

ZSTD

) to the

RecordBatchStream

constructor. Recommended next steps: * Open a GitHub Issue: Submit a feature request referencing your findings. Mention that for high-latency or bandwidth-constrained environments, the 0.9x "compression" ratio (uncompressed overhead) is a bottleneck. * Community Contribution: Since you have already identified the specific code location (

be/src/service/service_be/arrow_flight_sql_service.cpp

), you might consider submitting a Pull Request. Adding a session variable (e.g.,

arrow_flight_compression_codec

) to control this would be the standard StarRocks approach for such a feature. Note that while Arrow Flight is designed for high performance, StarRocks' current implementation prioritizes the "zero-copy" path. Compression adds a CPU-bound serialization step that may offset gains in high-bandwidth environments, which is likely why it wasn't enabled by default in the initial release. References • unloading/arrow_flight.md • administration/management/BE_configuration.md • loading/loading_introduction/feature-support-loading-and-unloading.md

GitHub issue created: [Feature] Arrow Flight SQL: support Arrow IPC compression (LZ4/ZSTD) for DoGet responses · Issue #73876 · StarRocks/starrocks