@Rocky why do I sometimes get these errors on CN scaledown? ------------------|------------------------------------------------------------------------------------------ 1 starrocks-sql widget_visibility_perf_summary_go_get_m1__payout_component_id_1000_MOSAIC: ProgrammingError(1064, 'Compute node not found. Check if any compute node is down. nodeId: -1 compute node: ') 12 starrocks-sql commissions_perf_summary_objective_table_UPDATED: ProgrammingError(1064, 'Compute node not found. Check if any compute node is down. nodeId: -1 compute node: ') 4 starrocks-sql commissions_perf_summary_chart_ytd_pct_to_plan_MOSAIC: ProgrammingError(1064, 'Compute node not found. Check if any compute node is down. nodeId: -1 compute node: ') 2 starrocks-sql commissions_transactions_all_rows_MOSAIC: ProgrammingError(1064, 'Compute node not found. Check if any compute node is down. nodeId: -1 compute node: ')

This error typically occurs during CN (Compute Node) scale-down because the FrontEnd (FE) attempts to schedule or execute a query fragment on a node that has been removed from the cluster's active node list. When you scale down by simply dropping a node (e.g., using

ALTER SYSTEM DROP COMPUTE NODE

), the node is immediately removed from the FE's metadata. Any in-flight queries or new queries being scheduled at that exact moment will fail with Error 1064, often showing

nodeId: -1

if the scheduler can no longer find a valid mapping for the query fragments. How to Prevent This To avoid these errors during scale-down, you should use the Graceful Exit mechanism (available in StarRocks v3.3 and later). 1. Use the Graceful Shutdown Script Instead of dropping the node via SQL or stopping the process abruptly, use the

-g

(graceful) flag:

bash
# On the CN node being scaled down
./bin/stop_cn.sh -g --timeout 60

• What it does: The CN marks itself as "exiting" and stops accepting new query fragments but continues to process existing fragments until they complete or the timeout is reached. • FE Awareness: From v3.4 onwards, the FE correctly identifies the

SHUTDOWN

state and will stop routing new queries to that node while allowing existing ones to finish. 2. Configure Wait Time You can control how long the CN waits for in-flight queries to finish by adjusting the following configuration in your `cn.conf`: • `loop_count_wait_fragments_finish`: The CN waits for

loop_count_wait_fragments_finish * 10

seconds. (Default is 2, which is 20 seconds). • For long-running queries, increase this value (e.g., set it to

6

for a 60-second wait). 3. Verify Node Status Before physically removing the node or its container, you can check its status in StarRocks to ensure it has finished its tasks:

sql
SHOW COMPUTE NODES;

Look for the

StatusCode

column: • `OK`: Running normally. • `SHUTDOWN`: Graceful exit in progress (safe for existing queries to finish). • `DISCONNECTED`: The node has fully exited and can be safely removed from infrastructure. Summary of Recommendations * Upgrade to v3.4+: If you are on an earlier version, upgrading provides much better FE/CN coordination during scaling. * Avoid

ALTER SYSTEM DROP

for active nodes: Use the graceful shutdown script first. Once the node is stopped and shows as

Alive: false

, you can then run the

DROP

command to clean up metadata if necessary. * Kubernetes Users: If you are using the StarRocks Operator, ensure you are using a recent version that handles the

preStop

hook with the

--graceful

flag automatically. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • faq/Deploy_faq.md • administration/Meta_recovery.md

@Rocky the stock cn_prestop.sh does not have the

--timeout 60

part you suggested above. Is that a problem? https://github.com/StarRocks/starrocks/blob/main/docker/dockerfiles/be/cn_prestop.sh#L7

The

cn_prestop.sh

script in the official StarRocks Docker repository is designed to notify the Frontend (FE) to stop routing new queries to a Compute Node (CN) before the container is terminated. While the "stock" script works for general purposes, omitting a timeout (specifically for the

curl

command or as part of the Graceful Exit process) can lead to the following issues: 1. Robustness against FE Unreachability The

curl

command in the stock script lacks a connection timeout (e.g.,

--connect-timeout 10

or

-m 60

). * The Problem: If the FE is unresponsive or network-partitioned during a scale-down or update,

curl

may hang indefinitely. * The Result: The Kubernetes

preStop

hook will block until the

terminationGracePeriodSeconds

(default 30s) expires, at which point the container is force-killed. Adding a timeout to the

curl

command ensures the script fails quickly and allows the shutdown process to proceed rather than hanging on a dead FE. 2. Transition to "Graceful Exit" (StarRocks 3.3+) From StarRocks v3.3 onwards, and enhanced in v3.4+, the recommended way to stop a node is the Graceful Exit mechanism using the

-g

flag in the stop scripts. In newer versions, instead of just calling the

deactivate

API via

curl

, you should ideally call:

bash
./bin/stop_cn.sh -g --timeout 600

* Why the timeout matters here: In this context,

--timeout

defines how long the CN process will wait for in-flight query fragments to finish before force-quitting. * Stock Script vs. Modern Approach: The stock

cn_prestop.sh

you linked uses an older method (API-based deactivation). While it stops new queries from being assigned, it doesn't provide the same coordinated "wait loop" for existing queries that the v3.4+

stop_cn.sh -g

command offers. Is it a problem? * For older versions (< 3.3): It is not a "problem," but adding

--connect-timeout 60

to the

curl

command is still a best practice for network reliability. * For newer versions (3.3+): If you want to ensure that long-running queries finish successfully during a rolling upgrade, the stock script's reliance on a simple API call is less "graceful" than using

stop_cn.sh -g --timeout <seconds>

. Summary of Recommendation If you are managing your own Docker/K8s environment, it is highly recommended to update your

preStop

hook or

cn_prestop.sh

to: 1. Add a request timeout to the curl command:

curl --connect-timeout 5 -m 10 ...

to prevent hanging on a dead FE. 2. Use Graceful Exit (v3.4+): If on a modern version, call

stop_cn.sh -g --timeout 60

to allow the CN to drain its own tasks locally before the container exits. Without these, your cluster is more susceptible to "Terminating" pods hanging or queries being killed abruptly during routine maintenance. References • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_COMPUTE_NODES.md • administration/stargo.md

@Rocky what does starlet_fs_read_prefetch_enable do?

The configuration parameter

starlet_fs_read_prefetch_enable

is a Back-end (BE) / Compute Node (CN) setting used in StarRocks shared-data (cloud-native) architecture. What it does: It enables read prefetching at the Starlet filesystem layer. Starlet is the internal abstraction layer that StarRocks uses to communicate with remote storage (such as AWS S3, Google Cloud Storage, Azure Blob Storage, or HDFS). When this parameter is set to `true`: 1. Latency Hiding: The Starlet client proactively fetches data from the remote object storage into a local buffer before the query execution engine explicitly requests it. 2. Improved Throughput: By pre-loading data, it helps mitigate the high network latency typically associated with object storage. This ensures that the CPU remains busy processing data rather than waiting for I/O operations to complete. 3. Sequential Read Optimization: It is particularly effective for large-scale sequential scans (e.g., full table scans or large range queries) where the next block of data to be read is predictable. Related Parameters: If you are tuning this for performance, you may also encounter these related settings: *

starlet_fs_stream_buffer_size_bytes

: Defines the size of the buffer used for these streaming/prefetching reads (commonly set to

1048576

or 1MB by default). *

starlet_use_star_cache

: Determines whether the local data cache (Data Cache) is enabled to store these blocks on local disk/SSD after they are read. Use Case: You would typically ensure this is enabled in a shared-data environment where your primary storage is on the cloud (object storage). It is usually enabled by default in recent versions of StarRocks to ensure optimal performance for cloud-native tables. References • administration/management/BE_configuration.md • unloading/Spark_connector.md • deployment/shared_data/gcs.md • deployment/shared_data/s3.md • deployment/shared_data/feature-support-shared-data.md • using_starrocks/caching/block_cache.md

Did

starlet_fs_stream_buffer_size_bytes

get renamed to ake_compaction_stream_buffer_size_bytes?

No,

starlet_fs_stream_buffer_size_bytes

was not renamed to

lake_compaction_stream_buffer_size_bytes

. They are two different parameters that control buffer sizes for different types of I/O operations in a shared-data (cloud-native) environment. Here is the breakdown of the difference: 1.

starlet_fs_stream_buffer_size_bytes

* Purpose: Controls the buffer size for general query prefetching. * Usage: When

starlet_fs_read_prefetch_enable

is set to

true

, this parameter determines the size of the memory buffer used to pre-fetch data from remote storage (S3/OSS/HDFS) during a query scan. * Scope: Primarily affects query performance by reducing the impact of remote storage latency. * Default: Historically

131072

(128KB), but often tuned to

1048576

(1MB) or higher for better performance in high-latency environments. 2.

lake_compaction_stream_buffer_size_bytes

* Purpose: Specifically controls the buffer size for compaction tasks in shared-data clusters. * Usage: It determines the remote I/O buffer size used when the "Lake" (cloud-native) engine reads data from object storage during background compaction. * Scope: Affects how quickly and efficiently the background compaction process can merge files. * Default:

1048576

(1MB). * Introduced: This parameter was explicitly added in v3.2.3 to allow users to tune compaction performance separately from query performance. Why you might be seeing them together: In StarRocks v3.2 and later, the development team began specializing configuration parameters to provide more granular control. • If you want to accelerate queries fetching data from S3, you tune the

starlet_fs_...

parameters. • If you want to accelerate background compaction (to prevent version accumulation or "too many versions" errors), you tune the

lake_compaction_...

parameters. Summary Table: Parameter | Function | Target Operation

starlet_fs_stream_buffer_size_bytes

| Prefetch Buffer | Queries / General Scans

lake_compaction_stream_buffer_size_bytes

| Remote I/O Buffer | Compaction / Background Merging If you are seeing

starlet_fs_stream_buffer_size_bytes

missing from the current official documentation, it is likely because it is now considered an "advanced" or internal Starlet setting, whereas

lake_compaction_...

is a documented tuning knob for managing cluster stability. References • administration/management/BE_configuration.md

@Rocky I can't find documentation link to

starlet_fs_stream_buffer_size_bytes

You are correct. You won't find an official documentation link for

starlet_fs_stream_buffer_size_bytes

because it is currently classified as an "internal" or "advanced" configuration parameter. While it is actively used in the StarRocks engine (especially in shared-data/cloud-native versions), it has not been promoted to the public-facing BE Configuration list. Key Facts about this Parameter: * Status: Valid but undocumented. * Role: It defines the memory buffer size (per stream) used by the Starlet filesystem layer when reading data from remote object storage (S3, OSS, HDFS). * Default Value:

131072

(128 KB). * Usage: It works in conjunction with

starlet_fs_read_prefetch_enable

. When prefetching is on, this buffer is filled in the background to hide storage latency. Why you might need it: If you are experiencing slow query performance on a shared-data cluster where data resides on object storage (like AWS S3 or MinIO), you might see recommendations from support or the community to increase this value. * Recommended Value: Often tuned to

1048576

(1 MB) or

10485760

(10 MB) to provide smoother I/O throughput in high-latency network environments. Related Documented Parameter: Since StarRocks v3.2.3, the engineering team has started separating query I/O from background I/O. If you are looking for a similar setting that is documented, see: * `lake_compaction_stream_buffer_size_bytes`: This controls the buffer size specifically for compaction tasks in a shared-data cluster. It was added to give users a documented way to tune background performance without affecting the internal Starlet defaults used for queries. Summary: If you need to change

starlet_fs_stream_buffer_size_bytes

, you can still add it to your

be.conf

and restart the BE/CN; it will take effect even without being in the official documentation. References • administration/management/BE_configuration.md • administration/management/FE_configuration.md

@Rocky would a CN that is currently not balanced with tablets yet still prefetch when

starlet_fs_read_prefetch_enable

is true?

The short answer is yes, but only if that CN is actually assigned work to perform. Whether a Compute Node (CN) prefetches data depends on whether it is performing a read operation from remote storage. The state of "tablet balance" affects what work is assigned to the CN, but it does not disable the prefetching logic itself. Here is the detailed breakdown of how this works in different scenarios: 1. For Internal Shared-Data (Cloud-Native) Tables In a shared-data cluster, tablets are logically "assigned" to specific CNs by the Frontend (FE). The FE tries to schedule query fragments to the CN that "owns" the tablet to maximize local cache hits. * If the CN has NO tablets assigned: The FE will likely not send any scan fragments for internal tables to this node. Since the CN isn't reading any data, the prefetcher stays idle. * If the CN is "unbalanced" (has fewer tablets): For the few tablets it does have, it will perform reads from S3/remote storage. When it does so,

starlet_fs_read_prefetch_enable=true

will trigger and it will prefetch data into its memory buffer to speed up those specific reads. * During Rebalancing: While tablets are being moved to the new CN, or if the FE decides to schedule a "remote scan" on a CN that doesn't yet own the data (to handle high load), the CN will read from remote storage and will use prefetching. 2. For External Tables (Data Lake / Catalogs) If you are using the CN to query data from a Data Lake (e.g., Hive, Iceberg, or Hudi on S3/HDFS/GCS): * Tablet balance does not matter: External tables do not use the same tablet-to-worker "assignment" logic as internal tables. Scan fragments for data lake queries are typically distributed across all available CNs in the cluster. * Prefetching is active: Even if a CN has zero internal tablets assigned, it will be fully utilized for data lake queries. In this case,

starlet_fs_read_prefetch_enable

is highly beneficial and will be active for every scan fragment the CN processes. 3. Starlet Layer Logic The parameter

starlet_fs_read_prefetch_enable

exists at the Starlet Filesystem layer. This layer is agnostic to "tablets" or "balancing." _ If the execution engine says: _"Open file X on S3 and read 1MB,"* Starlet checks if prefetching is enabled. * If

true

, it begins fetching subsequent blocks of that file into memory while the engine processes the first block. * It does this regardless of why the CN was asked to read the file (whether it's an internal tablet, an external Parquet file, or a compaction task). Summary Scenario | Will it prefetch? | Reason New CN with 0 internal tablets | No (for internal tables) | It won't be scheduled any scan tasks for those tables. New CN with 0 internal tablets | Yes (for external tables) | Work is distributed across all nodes; prefetching will trigger on any remote read. Unbalanced CN (few tablets) | Yes | It will prefetch for any scan task it receives for its assigned tablets. During Tablet Migration | Yes | The destination CN will prefetch data as it "warms up" or reads the remote segments. Recommendation: If you are adding a new CN and want it to be productive immediately for external queries or to speed up the processing of the tablets it is receiving, keep

starlet_fs_read_prefetch_enable

set to

true

. It ensures that whenever the CN is called to action, it performs I/O as efficiently as possible. References • table_design/table_design.md • data_source/file_external_table.md • data_source/External_table.md

@Rocky is there any risks to

starlet_fs_stream_buffer_size_bytes

being increased?

Increasing

starlet_fs_stream_buffer_size_bytes

can significantly improve scan performance from remote storage (like S3 or HDFS), but there are a few key risks you should be aware of: 1. Multiplicative Memory Consumption (The OOM Risk) The most critical risk is that this buffer is per stream, not a global pool. * The Math: Total Memory =

starlet_fs_stream_buffer_size_bytes

× Number of Concurrent Read Streams. * The Consequence: If you have many concurrent queries, each with multiple scan threads, your memory consumption can spike rapidly. If you set this to 10MB and have 100 concurrent scan streams, that's an extra 1GB of memory. In high-concurrency environments, this is a common cause of BE (Backend) or CN (Compute Node) OOM (Out of Memory) crashes. 2. Reduced Memory for Other Operations StarRocks has a hard memory limit (

mem_limit

, usually 90% of total RAM). Every byte allocated to the Starlet prefetch buffer is a byte taken away from: * Join Hash Tables: Large joins might fail or trigger "spill to disk" earlier. * Aggregation Buffers: Group-by operations might become slower. * Metadata Caching: Less room for tablet or segment metadata. 3. Memory Waste (Internal Fragmentation) If the value is set too high (e.g., 64MB) but your data segments are small or your queries only read small portions of files (e.g., selective filters on Parquet), you are allocating large chunks of memory that may never be fully utilized. This "internal fragmentation" effectively lowers the overall memory efficiency of the node. 4. CPU and Network Pressure While the buffer helps hide latency, filling a much larger buffer at the start of every stream can cause: * Network Spikes: Sudden bursts of high network traffic as many streams try to fill their large prefetch buffers simultaneously. * Context Switching: Managing more large memory allocations can increase overhead on the system's memory allocator (jemalloc). Recommendations for Safe Tuning: If you decide to increase it, follow these best practices: 1. Start Small: Don't jump from 128KB to 100MB. Try 1MB (

1048576

) or 4MB (

4194304

) first. 2. Monitor "Process Mem": Use the StarRocks metrics or

top/htop

to watch the memory usage of the

starrocks_be

process during peak query load. 3. Check Concurrency: Verify your

parallel_fragment_exec_instance_num

and expected query concurrency. 4. Use v3.2.3+ Logic: If your goal is to speed up compaction rather than queries, leave

starlet_fs_stream_buffer_size_bytes

at the default and only increase

lake_compaction_stream_buffer_size_bytes

. This isolates the risk to background tasks rather than user-facing queries. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md • faq/Others.md • loading/loading_introduction/loading_considerations.md • faq/operation_maintenance_faq.md

@Rocky default seems to already be 1MB HERE NAME LIKE '%starlet_fs_stream_buffer_size_bytes%'; +-------+-------------------------------------+---------+---------+---------+-------+ | BE_ID | NAME | VALUE | DEFAULT | MUTABLE | TYPE | +-------+-------------------------------------+---------+---------+---------+-------+ | 94451 | starlet_fs_stream_buffer_size_bytes | 1048576 | 1048576 | 1 | int32 | | 10001 | starlet_fs_stream_buffer_size_bytes | 1048576 | 1048576 | 1 | int32 | +-------+-------------------------------------+---------+---------+---------+-------+

You are absolutely right. I appreciate you catching that! In older versions of StarRocks (and early shared-data builds), the default was indeed 128 KB (

131072

). However, as the shared-data (Cloud-Native) architecture matured, the engineering team bumped the default to 1 MB (

1048576

) to better handle the higher latency of object storage like S3. What this means for you: * Default is 1MB: In your current version, StarRocks is already allocating 1MB per read stream. * Is it enough? For most standard workloads, 1MB is a good "sweet spot." * When to increase it further: If you are running on a network with extremely high latency or very high bandwidth (like 10Gbps+ links to your storage) and you see that your scan speed is significantly lower than your network capacity, you might try 4MB or 8MB. * Memory Impact: Since the default is already 1MB, your memory baseline is higher than in older versions. If you decide to increase it to, say, 10MB, the risk of OOM (Out of Memory) increases tenfold because that buffer is multiplied by every active scan thread. Summary of common values: Version Era | Internal Default | Purpose Early / Legacy | 128 KB | Minimal memory footprint for local disk I/O. Current (v3.x) | 1 MB | Optimized for object storage (S3/OSS/HDFS) throughput. Aggressive Tuning | 4 MB - 16 MB | For high-throughput requirements where RAM is plentiful. Pro-tip: Since the parameter is

MUTABLE

(as seen in your screenshot), you can test a higher value on a single node without a restart using:

sql
ADMIN SET FRONTEND CONFIG ("starlet_fs_stream_buffer_size_bytes" = "4194304"); 
-- Note: This is a BE config, use the appropriate method for your deployment 
-- or update be.conf for a permanent change.

(Actually, for BE configs, you would typically use

curl -X POST http://<be_ip>:<be_http_port>/api/update_config?starlet_fs_stream_buffer_size_bytes=4194304

if you want to change it dynamically without a restart.) References • administration/management/BE_configuration.md • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • loading/loading_introduction/loading_considerations.md • table_design/table_design.md • mirrorship.cn - StarRocks 3.0-rc01存算分离测试报告 • mirrorship.cn - StarRocks 存算分离性能测试报告 References • administration/management/BE_configuration.md • administration/management/FE_configuration.md