@Rocky OOM reading ADLS Gen2 (abfss) parquet — StarRocks 3.5.17, shared-data We're running a parallel backload that reads an external table (Unified catalog) backed by ADLS Gen2 (

abfss://

). Many queries fail with:

fail to hdfsPread abfss://.../<file>.parquet:
error=Resource temporarily unavailable,
root_cause=OutOfMemoryError: Java heap space

Client sees `ERROR 1064 (HY000)`; audit log records

errorCode=IO_ERROR

. Our observations: • It's a

Java heap space

OOM, and the failing call is

hdfsPread

. • The CN it failed on was at ~29% of a 50 GB`mem_limit`, no restart — so the node itself wasn't out of RAM. • It happens under high read concurrency (many simultaneous range-scan queries); single queries seem fine. What we're unsure about / questions: 1. For

abfss

in 3.5.x, does the read path go through libhdfs + a JVM (

hadoop-azure

), or is there a native reader? We're inferring it's JVM-based only from the`hdfsPread`name and the "Java heap space" message — can someone confirm? 2. If it is JVM-based: which heap is this, and how should it be sized/capped? Our`cn.conf`sets no explicit`-Xmx`. 3. Is there a config to bound the per-read buffer / read-ahead size or connector concurrency, rather than just enlarging the heap? 4. Are large parquet row-group sizes on the writer side a known trigger for big single

hdfsPread

allocations? Running 3.5.17, shared-data (FE + CN).

The

OutOfMemoryError: Java heap space

during

hdfsPread

indicates that your

abfss

read path is utilizing the JNI-based HDFS client (libhdfs) which interacts with a JVM running inside the CN process. 1. Does

abfss

use a native reader? While StarRocks 3.x has a native C++ Parquet reader, the filesystem abstraction for

abfss://

(and

hdfs://

) still relies on the Hadoop client via JNI (libhdfs) in many configurations. The error

hdfsPread

is a direct call from StarRocks C++ code to the Java Hadoop client (

hadoop-azure

). The OOM occurs within this JVM. 2. Which heap is this and how to size it? This is the embedded JVM heap within the CN process. By default, if

-Xmx

is not specified in

cn.conf

, the JVM uses its own ergonomic defaults (often 1/4 of the physical RAM, which may not align with StarRocks'

mem_limit

). * How to set it: Add or modify the

JAVA_OPTS

environment variable in your `cn.conf`:

bash
    # Example: Setting heap to 8GB.
    # Ensure this + CN mem_limit fits within physical RAM.
    JAVA_OPTS="-Xmx8G -Xms8G"

* Why CN memory was low: The CN

mem_limit

(50GB) tracks C++ memory usage. The JVM heap is managed separately by the Java Runtime. C++ is "unaware" of the Java heap internal exhaustion, which is why the CN reported only 29% usage while the JVM crashed. 3. Configs to bound memory / concurrency Under high concurrency, each scan thread can allocate its own buffers. You can tune the following: *

hdfs_read_buffer_size_kb

(FE Configuration): This controls the buffer size StarRocks requests for each HDFS/ABFSS read. * Default: 8192 (8 MB). * Action: If you have many concurrent scans, try reducing this to

1024

or

2048

to lower the per-thread memory pressure.

sql
    ADMIN SET FRONTEND CONFIG ("hdfs_read_buffer_size_kb" = "2048");

* Hedged Reads: Ensure

hdfs_client_enable_hedged_read

is set to

false

(default) in

cn.conf

. Enabling this creates additional threads and buffers in the JVM, significantly increasing heap pressure. 4. Parquet row-group sizes and

hdfsPread

Yes, large Parquet row groups (e.g., 256MB or 512MB) are a known trigger. * When StarRocks performs a range scan, it may attempt to read large contiguous blocks. * If the reader cannot skip data (e.g., no predicate pushdown or late materialization is ineffective), it pulls large chunks through

hdfsPread

. * The JNI layer must allocate these buffers in the Java heap to pass them to C++. With high concurrency, these allocations can quickly exhaust a default heap. Recommendations 1. Explicitly set the JVM Heap: Add

JAVA_OPTS="-Xmx8G"

(or larger, depending on your node's total RAM) to

cn.conf

. 2. Reduce Buffer Size: Set

hdfs_read_buffer_size_kb

to

2048

on the FE. 3. Verify Native Reader: Check if

enable_native_reader

is

true

(default in 3.x). Note that "native reader" refers to the Parquet parsing logic; the underlying IO for

abfss

may still be JNI-based. 4. Monitor with `jstat`: You can attach to the CN's embedded JVM using

jstat -gcutil <cn_pid> 1000

to watch the heap (

O

column for Old Gen) during your backload. References • administration/Meta_recovery.md • faq/operation_maintenance_faq.md • data_source/datalake_faq.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md

any chance you are seeing this https://github.com/StarRocks/starrocks/issues/74653 ?

I think #74653 is about metadata scanning while my issue is related to read-buffer allocation