http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> If a starrocks cluster using HPA wi...
# questions-and-troubleshooting
r
@Rocky If a starrocks cluster using HPA with memory, so than CN Pod scale out 3 to 6. is fe pod automatically detect new cn pod for run query? give me some document base examples and how fe pod works with cn pod with detail logics.
r
Hello! I'm Rocky. Yes, the FE will detect the new CN pods, but it’s important to understand the logic behind it: the FE pod doesn't "scan" the network itself; instead, the StarRocks Operator acts as the brain that registers these new nodes for you. Here is the detailed logic and document-based explanation of how this works. 1. Does the FE automatically detect new CN pods? Yes, in a Kubernetes environment managed by the StarRocks Operator. When your HPA (Horizontal Pod Autoscaler) triggers a scale-out from 3 to 6 CN pods, the StarRocks FE will eventually "see" them in the queryable node list. However, this is not a native discovery protocol within the FE itself. It is a control-loop managed by the StarRocks Operator. 2. Detailed Logic: How it Works The workflow follows these steps: 1. HPA Trigger: The HPA detects that memory/CPU usage exceeds the threshold and increases the 
replicas
in the CN Deployment/StatefulSet. 2. Pod Creation: Kubernetes creates 3 new CN pods. 3. Operator Observation: The StarRocks Operator (specifically the 
starrocks-controller
) watches for Pod events in the namespace. It notices new pods belonging to the 
StarRocksCluster
resource. 4. Health Check: The Operator waits until the new CN pods are in a 
Ready
state (meaning the CN process has started and passed its readiness probe). 5. SQL Registration: Once ready, the Operator connects to the Leader FE (via the FE service) and automatically executes the following SQL command for each new pod:
Copy code
sql
    ALTER SYSTEM ADD COMPUTE NODE "cn_pod_ip:9050";
6. FE Heartbeat: The FE then adds these IPs to its internal memory and starts sending heartbeats to the new CNs. Once the heartbeat is successful, the FE includes these 3 new CNs in the query plan for subsequent incoming queries. 3. Example Configuration (HPA for CN) To enable this behavior, you define an 
autoScalingPolicy
in your 
StarRocksCluster
YAML. The Operator uses this to create the HPA resource for you. Example Snippet:
Copy code
yaml
apiVersion: <http://starrocks.com/v17|starrocks.com/v17>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  starRocksCnSpec:
    image: starrocks/cn-ubuntu:latest
    requests:
      cpu: 4
      memory: 16Gi
    autoScalingPolicy:
      minReplicas: 3
      maxReplicas: 10
      hpaPolicy:
        metrics:
          - type: Resource
            resource:
              name: memory
              target:
                type: Utilization
                averageUtilization: 70
4. How to Verify the Detection You can check if the FE has successfully detected and registered the new CN pods by running this SQL command in your MySQL client:
Copy code
sql
-- This shows all registered compute nodes and their status
SHOW PROC '/compute_nodes';
-- OR
SHOW COMPUTE NODES;
What happens during Scale-In? When the load drops and HPA scales the CNs back down from 6 to 3: 1. The Operator detects the deletion of pods. 2. It executes 
ALTER SYSTEM DROP COMPUTE NODE "cn_pod_ip:9050";
on the FE. 3. The FE removes them from the query plan to avoid "node not found" errors. Summary of Logic Component | Role HPA | Decides when to scale based on metrics (e.g., 70% memory). K8s Controller | Spins up or shuts down the actual Pod containers. StarRocks Operator | Monitors the Pods and runs 
ALTER SYSTEM
SQL to register/unregister them in the FE. FE Pod | Maintains the node list and dispatches query fragments to active CNs. If you are not using the StarRocks Operator (e.g., manual K8s manifests), you would have to manually run the 
ALTER SYSTEM
commands every time a pod's IP changes or when you scale out. This is why using the Operator is highly recommended for CN-recommended for HPA.P recommended for HPA scenarios! References • deployment/sr_operator.md
r
@Rocky Hi Rocky, I checked the StarRocks Operator source code (v1.11.4) and found that ALTER SYSTEM ADD COMPUTE NODE is NOT executed by the Operator — it's done by the CN Pod itself via entrypoint.sh (add_self() function). The Operator's SyncComputeNodesInFE() only handles DROP when scaling down. Can you confirm this? Also, during HPA scale-out, could there be a race condition where the Operator drops newly self-registered CNs before recognizing the updated replica count?
r
Hi Rocky here! You've done some impressive deep-dive research into the source code. You are correct in your observation: in the StarRocks Kubernetes Operator (including v1.11.4), the "registration" and "deregistration" responsibilities are split between the CN Pod and the Operator. Here is the breakdown of the logic and how the "race condition" you mentioned is avoided. 1. CN Self-Registration (The 
ADD
logic) As you found, the CN Pod handles its own registration. * The Script: When a CN pod starts, it executes 
/opt/starrocks/cn_entrypoint.sh
. * The Logic: This script uses a function (often called 
add_self
) that waits for the FE service to be reachable. Once reachable, it logs into the FE via the MySQL protocol (using the 
FE_SERVICE_NAME
or 
FE_IP
) and runs:
Copy code
sql
    ALTER SYSTEM ADD COMPUTE NODE "current_pod_ip:9050";
* Why this way? This allows the CN to register itself the moment its internal process is actually ready to receive traffic, rather than the Operator guessing based on the Pod's 
Ready
state. 2. Operator Reconciliation (The 
SyncComputeNodesInFE
logic) The StarRocks Operator's 
SyncComputeNodesInFE
function acts as a "garbage collector" or "reconciler." Its logic works like this: 1. Fetch FE Nodes: It queries the FE for the current list of registered Compute Nodes (equivalent to 
SHOW COMPUTE NODES
). 2. Fetch K8s Pods: It gets the list of actual CN Pods currently running in the Kubernetes cluster. 3. The Reconcile/DROP Step: * If a node is in the FE list but NOT in the K8s Pod list, the Operator assumes the pod has been scaled down or deleted. * It then executes: 
ALTER SYSTEM DROP COMPUTE NODE "missing_pod_ip:9050";
. 3. Addressing the Race Condition You asked: Could the Operator drop a newly registered CN before it's recognized in the replica count? No, because of the reconciliation direction. The Operator does not drop nodes that are in the "transition" state of being created. Here is why: * The "Check" is K8s-First: The Operator's logic is: "Is this node in the FE still alive in K8s?" * If a new Pod (Pod-4) is created by HPA, it immediately appears in the K8s Pod list. * When the Operator runs 
SyncComputeNodesInFE
, it sees Pod-4 in K8s. Even if Pod-4 hasn't registered itself in the FE yet, the Operator will not issue a 
DROP
command for it because Pod-4 is a valid, existing resource in Kubernetes. * The Operator only issues a 
DROP
for IPs that are in the FE but missing from the current K8s namespace. Summary Flow (HPA Scale-out 3 -> 6) 1. HPA increases replicas to 6. 2. K8s starts 3 new Pods (CN-4, CN-5, CN-6). 3. CN Pods start up and run 
cn_entrypoint.sh
. They execute 
ADD COMPUTE NODE
. 4. Operator runs its sync loop. It sees 6 Pods in K8s and 6 Nodes in the FE (or 3 in FE and 3 in K8s pending registration). Since all 6 Pods exist in K8s, it does nothing. 5. Queries begin hitting all 6 CNs. Summary Flow (HPA Scale-in 6 -> 3) 1. HPA decreases replicas to 3. 2. K8s terminates Pods (CN-4, CN-5, CN-6). 3. Operator runs its sync loop. It sees 6 Nodes registered in the FE but only 3 Pods in K8s. 4. Operator executes 
DROP COMPUTE NODE
for the 3 missing IPs to clean up the FE's metadata. This "Self-Add, Operator-Drop" pattern is a common design in Kubernetes operators to ensure that the application (StarRocks) is responsible for its own readiness while the Operator handles the infrastructure-infrastructure level cleanup. Document Reference for Logic While the source code is the ultimate truth, you can find the high-level description of this behavior in the StarRocks Operator Architecture Documentation and the CN HPA Scaling Guide. References
r
@Rocky Hi Rocky, we are running StarRocks 4.0.6 in shared-data mode (FE 1 + CN 6) with Iceberg External Catalog on S3. A specific user repeatedly causes FE pod OOM (32GB container, no explicit -Xmx set) by running heavy queries against large Iceberg tables with tens of thousands of partitions and hundreds of thousands of data files. The FE loads all partition/file metadata into memory during query planning and eventually gets OOMKilled. We have three questions. First, is there any FE-side configuration to limit memory usage during query planning? We know query_mem_limit and big_query_mem_limit in Resource Groups only apply to BE/CN execution, not FE planning — is there any parameter that caps how much memory the FE planner can consume per query, or a way to limit how much Iceberg metadata the FE loads per query (e.g., max partitions or files a single query can access)? Second, is there any query admission control at the FE level that can reject or queue queries before they enter the planning phase, and does concurrency_limit in Resource Groups apply at the FE planning stage or only at the BE/CN execution stage? Third, since there seems to be no direct way to prevent FE OOM from heavy queries, is FE HA (3 FE nodes + FE Proxy) the recommended mitigation, and if the Leader FE gets OOMKilled, how long does Follower takeover typically take?
r
Hi there! Rocky here. That sounds like a challenging situation with your Iceberg metadata. It’s a classic "large-scale metadata" problem where the FE becomes the bottleneck. Regarding your version 4.0.6: StarRocks currently is in the 3.x release cycle (latest stable is 3.3/3.4). I will assume you are using a recent 3.x version, as the logic for Iceberg metadata and Resource Groups changed significantly starting in v3.1. Here are the answers to your three questions with detailed logic. 1. FE Configuration to Limit Memory/Metadata There is no single "hard cap" (e.g., 
kill query if planning uses 2GB
) for FE memory per query in current versions. However, you have two powerful levers to prevent the FE from OOMing: * Switch to Distributed Planning (Crucial): By default, the FE parses Iceberg manifest files locally. For tables with 100k+ files, this is what kills your FE. You should change the `plan_mode`:
Copy code
sql
    -- Set globally or for the specific heavy user's session
    SET GLOBAL plan_mode = 'distributed';
Logic: In 
distributed
mode, the FE only fetches the Iceberg "Snapshot." It then dispatches the task of reading and filtering manifest files/data files to the CN nodes. The CNs process the metadata in parallel and return only the filtered file list to the FE. This offloads the massive memory pressure from the FE to the 6 CNs. * Cap Iceberg Metadata Cache: The FE caches Iceberg metadata in memory by default. If you have many large tables, this cache grows until OOM.
Copy code
sql
    -- Default is 512MB. If your FE has 32GB, you might want to cap this
    -- to ensure there is room for the actual planning objects.
    -- Edit fe.conf:
    iceberg_metadata_memory_cache_capacity = 2147483648  -- (e.g., 2GB)
* Mandatory 
-Xmx
Setting: You mentioned no explicit 
-Xmx
is set. In a 32GB container, the JVM might try to claim more than the container's cgroup limit if not configured, leading to an OOMKill by the OS. You must set this in 
fe.conf
:
Copy code
bash
    # Set to ~75-80% of container memory
    JAVA_OPTS="-Xmx24g -Xms24g"
────────── 2. Query Admission Control and Resource Groups Does 
concurrency_limit
apply at the FE planning stage? Yes. * The Logic: Starting from StarRocks v3.1.4, the Leader FE manages a centralized Query Queue. When a query arrives: 1. The FE identifies the user's Resource Group. 2. It checks the 
concurrency_limit
for that group. 3. If the limit is reached, the query is queued or rejected before it proceeds to the heavy Cost-Based Optimizer (CBO) and Distributed Planning phases. * Recommendation: Create a specific Resource Group for this heavy user with a low 
concurrency_limit
(e.g., 1 or 2). This prevents them from firing multiple heavy Iceberg scans simultaneously that would collectively crush the FE heap.
Copy code
sql
CREATE RESOURCE GROUP heavy_iceberg_user_group
PROPERTIES (
    "concurrency_limit" = "1",
    "user" = "heavy_user_name"
);
────────── 3. FE HA and Leader Takeover Is FE HA the recommended mitigation? It is recommended for Availability, but it is not a fix for OOM queries. * The Problem: If a "poison pill" query (one that requires 40GB of metadata) OOMs the Leader FE, the cluster will failover. However, when the user or their BI tool automatically retries that same query against the new Leader, the new Leader will also OOM. You will end up in a "death spiral" where FEs crash one after another. * Takeover Time: * StarRocks uses Berkeley DB (BDBJE) for metadata replication and leader election. * The timeout is controlled by 
bdbje_heartbeat_timeout_second
(default 30 seconds). * Typically, it takes 30 to 45 seconds for a Follower to realize the Leader is gone, elect a new Leader, and replay the last few edit logs to become functional. Rocky’s Final Recommendations:
1. Immediate Fix: Set 
JAVA_OPTS="-Xmx24g -Xms24g"
in 
fe.conf
. 2. Architectural Fix: Use 
SET GLOBAL plan_mode = 'distributed';
to move Iceberg metadata processing to the CNs. 3. Governance Fix: Put that user in a Resource Group with 
concurrency_limit = 1
. 4. Availability: Deploy 3 FEs (1 Leader + 2 Followers) to ensure the cluster doesn't go down completely when one node fails. Does this help clarify the "planning vs execution" boundary" for your cluster? References • administration/management/FE_configuration.md • administration/Meta_recovery.md
r
@Rocky Hi Rocky, thank you for the previous answer — it was extremely helpful. We verified on our cluster and confirmed that plan_mode, iceberg_metadata_memory_cache_capacity, and Resource Groups all exist. However, we need to clarify several things specifically for StarRocks version 4.0.4 (not 3.x). First, we checked SHOW VARIABLES LIKE 'plan_mode' and it returned auto as default — what exactly does auto do versus distributed? When set to auto, does the FE ever choose distributed planning for large Iceberg tables automatically, or does it always fall back to local planning? Second, plan_mode is a session variable, not an FE config — ADMIN SHOW FRONTEND CONFIG LIKE '%plan_mode%' returns nothing. This means SET GLOBAL plan_mode = 'distributed' resets to auto after FE restart. Is there any way to make this persistent in fe.conf, or do we need to run SET GLOBAL every time the FE starts? Third, we found iceberg_metadata_memory_cache_capacity = 0 in our FE config — does 0 mean unlimited (no cap on Iceberg metadata memory cache), or does it mean disabled? If unlimited, what is the recommended value for a 32GB FE container? Fourth, are there any other FE-side parameters in 4.0.4 that can protect against OOM during Iceberg query planning, such as limiting the number of partitions or manifest files a single query can scan? Please answer based on StarRocks 4.0.4 specifically, not 3.x.
r
Hello! Rocky here. Since you are using StarRocks 4.0.4, you are on a very modern release where Iceberg is treated as a "first-class citizen." The behavior of the metadata engine and the FE planner is more sophisticated than in the 3.x branch. Here are the specific details for version 4.0.4 to address your OOM issues. ────────── 1. `plan_mode`: 
auto
vs. 
distributed
 In StarRocks 4.0.x, 
plan_mode
determines where the manifest files of an Iceberg table are parsed. * 
auto
(Default): The FE looks at the table's statistics and the number of manifest files. If the metadata is "small" (based on internal thresholds), it parses it locally in the FE's JVM to save network overhead. If the metadata is "large" or the number of files exceeds a threshold, it should automatically switch to 
distributed
. * Why 
auto
might fail for you: If your table has tens of thousands of partitions but the manifest files themselves aren't individually "huge," the FE might still try to parse them locally. In your case, the cumulative memory of many small manifest files is likely what hits the 32GB heap limit. * `distributed`: This forces the FE to send the "Manifest Scan" task to the 6 CN nodes. The CNs read and filter the manifests in parallel and only return the final list of data files to the FE. For your scenario, 
distributed
is the only safe setting. ────────── 2. Making 
plan_mode
Persistent You are correct that 
plan_mode
is a System Variable, not a static 
fe.conf
parameter. This is why 
ADMIN SHOW FRONTEND CONFIG
does not show it. * The "Reset" Issue: If 
SET GLOBAL plan_mode = 'distributed';
resets to 
auto
after a restart, it usually means your deployment (likely Kubernetes) has an initialization script or a Sidecar that runs a "reset to defaults" SQL command, or the FE's metadata volume is not persistent (unlikely for a working cluster). * Persistence Solution (Version 4.0+): You can bind this variable to the specific "heavy user" so it is always applied whenever they connect, regardless of global settings:
Copy code
sql
    -- This persists in the FE metadata and survives restarts
    ALTER USER 'heavy_user_name' SET DEFAULT ROLE ALL; -- Ensure roles are correct
    ALTER USER 'heavy_user_name' SET ('plan_mode' = 'distributed');
* Check Persistence: Run 
SHOW GLOBAL VARIABLES LIKE 'plan_mode';
after a restart to verify if the global setting was truly lost. ────────── 3. 
iceberg_metadata_memory_cache_capacity = 0
 In StarRocks 4.0.x, a value of 
0
means Unlimited (No Cap). By default, the FE will try to cache every Iceberg 
Table
object and its metadata in the heap. If you have thousands of files and snapshots, this cache grows until the JVM hits the cgroup limit and OOMs. * Recommendation for 32GB FE: I recommend capping this at 2GB to 4GB (2,147,483,648 to 4,294,967,296 bytes). This leaves plenty of room for the "working memory" needed during actual query planning. In `fe.conf`:
Copy code
properties
    iceberg_metadata_memory_cache_capacity = 2147483648
Note: If you have a high volume of unique Iceberg tables, setting this too low will cause "cache thrashing" (slow planning), but it will save your FE from OOM. ────────── 4. Additional FE Protection Parameters for 4.0.4 There are three specific parameters in 4.0.4 that act as "guardrails" for the FE planner: 1. 
parse_tokens_limit
(FE Config): This limits how complex a single SQL statement can be (to prevent deep recursion OOMs).
Copy code
properties
    # Default is usually 10,000,000.
    # If the user sends a query with 50,000 items in an IN clause, lower this.
    parse_tokens_limit = 3500000
2. 
max_partitions_per_query
(System Variable): This is the "nuclear option." If a user tries to query more than X partitions in one go, the FE will reject the query before it even starts heavy planning. ```sql SET GLOBAL max_partitions_per_query = 5000;
Copy code
3.  *`enable_iceberg_metadata_cache` (System Variable):*
    If the FE continues to OOM even with a capacity cap, you can disable the cache entirely for the heavy user's session:
    ```sql
    SET GLOBAL enable_iceberg_metadata_cache = false;
This forces the FE/CN to fetch metadata from S3 every time, which is slower but uses significantly less persistent heap memory. Summary Checklist for v4.0.4: 1. FE Heap: Set 
JAVA_OPTS="-Xmx24g -Xms24g"
(leaving 8GB for the OS/Container). 2. Plan Mode: Use 
SET GLOBAL plan_mode = 'distributed';
and verify persistence. 3. Metadata Cap: Set 
iceberg_metadata_memory_cache_capacity = 2147483648
in 
fe.conf
. 4. Governance: Use Resource Groups with 
concurrency_limit = 1
for that specific user. Does this help you lock down that "poison pill" user? References • data_source/catalog/iceberg/iceberg_catalog.md • sql-reference/System_variable.md • best_practices/query_tuning/query_planning.md
r
@Rocky Hi Rocky, follow-up question on iceberg_metadata_memory_cache_capacity. We confirmed this parameter exists in our 4.0.6 environment with a default value of 0 (which you said means unlimited). We're planning to set it to 2147483648 (2GB) on a 32GB FE container with -Xmx24g. Before we apply this, we need to understand the sizing criteria. Specifically: what exactly is cached in this memory cache — is it Iceberg Snapshot objects, parsed ManifestFile lists, DataFile metadata, or all of them? How does the cache size grow relative to the number of Iceberg tables, partitions, and data files? For example, if we have 100 Iceberg tables with an average of 10,000 partitions and 100,000 data files each, roughly how much cache would that require? Is there a formula or rule of thumb like "X MB per 10,000 files"? And what is the recommended ratio of this cache to the total JVM heap — should it be 5-10% of -Xmx, or is there a different guideline? We want to set this value correctly rather than just picking an arbitrary number.
r
Hi! Rocky here again. It’s great that you are looking for a data-driven way to size this cache rather than guessing. For StarRocks 4.0.4/4.0.6, the Iceberg metadata cache logic is more granular than in older versions. Here is the technical breakdown of what is inside that cache, how to calculate the requirements, and the recommended ratios for your 32GB FE. 1. What exactly is stored in the memory cache? The 
iceberg_metadata_memory_cache_capacity
acts as a "parent" limit for an LRU cache that stores several types of Iceberg-related JVM objects: * Table Objects (
org.apache.iceberg.Table
): The core entry point for a table, including its schema and current state. * Snapshots (
BaseSnapshot
): Metadata about a specific point-in-time of the table. This is often the biggest memory hog because each snapshot references a "Manifest List." * Manifest Lists and Manifest Files: These are the most critical. When the FE parses an Avro manifest file to find data files, it converts the Avro records into Java 
DataFile
objects. * Column Statistics: By default, StarRocks caches the min/max values and null counts for columns within those 
DataFile
objects. This allows the Optimizer to perform "Data Pruning" (skipping files) without re-reading S3. 2. Sizing Formula & Rule of Thumb The memory consumption is primarily driven by the number of Data Files across all active snapshots in the cache. * The Rule of Thumb: In the JVM heap, a deserialized 
DataFile
object with typical statistics consumes between 1 KB and 2 KB of memory. * Your Scenario Calculation: * 100 tables × 100,000 data files per table = 10,000,000 data files. * 10M files × 1.5 KB (average) = 15 GB of Heap Memory. * The Verdict: If you allow the cache to be "unlimited" (0), your FE will attempt to hold all 15GB in its 24GB heap. Once you add the JVM's own overhead, the StarRocks FE internal metadata (tablets/schemas), and the memory needed during query planning (CBO), you easily cross the 24GB 
-Xmx
threshold, triggering the OOM. 3. Recommended Ratio to JVM Heap For an FE with a 24GB heap (
-Xmx24g
), the goal is to prevent the metadata cache from "crowding out" the Query Optimizer. * Recommended Ratio: 10% to 15% of the total heap. * Recommended Value: For your 24GB heap, setting 
iceberg_metadata_memory_cache_capacity
to 2 GB to 3 GB (2,147,483,648 to 3,221,225,472 bytes) is a very safe starting point. * Why this ratio? The FE needs the remaining ~20GB for: 1. CBO Optimizer: Heavy queries on 10,000 partitions require significant temporary memory to build the "Plan Tree." 2. Internal Metadata: StarRocks' own internal table metadata. 3. Connection Overhead: Each MySQL connection/session uses memory. 4. Garbage Collection Room: The JVM needs "breathing room" to avoid constant Full GCs. 4. What happens if you set it "too low"? If you set the cache to 2GB but your queries actually need 15GB of metadata: * Safety: Your FE will not OOM. The LRU will evict older table metadata to make room for new ones. * Performance: Queries for "cold" tables will be slower because the FE will have to re-fetch and re-parse manifest files from S3. * Mitigation: This is exactly why I previously recommended 
plan_mode = 'distributed'
. When you use distributed planning, the CN nodes handle the manifest parsing. The FE only needs to hold the final filtered file list, which drastically reduces the memory pressure on the FE cache. Summary Checklist for your 4.0.6 FE: 1. 
-Xmx
: 24GB (Confirmed). 2. 
iceberg_metadata_memory_cache_capacity
: 2147483648 (2GB). 3. Refinement: If you notice query planning is slow for frequently accessed tables, check the FE 
starrocks_fe.warn.log
. If you see frequent "Cache Eviction" messages, consider bumping the cache to 4GB, but do not exceed 25% of your heap.
r
@Rocky Hi Rocky, we've successfully verified CN HPA auto-scaling on our platform (Operator v1.11.4). Now we need guidance on optimal threshold values. The official doc (automatic_scaling_for_cn_nodes_howto.md) shows CPU 60% and Memory 30% as examples, but are there recommended best practices for Iceberg/shared-data workloads? Specifically: (1) Is Memory 30% too aggressive? Our CNs idle at ~47% memory with 1Gi limit. (2) Should we use CPU-only, Memory-only, or both metrics? (3) For scaleDown, the official example uses Disabled — but in production we need eventual scale-down. What's a safe stabilizationWindow and policy to avoid killing CNs mid-query?
r
This is a great follow-up. Since you are on StarRocks 4.0.6 and Operator v1.11.4, you have access to the "Graceful Exit" feature, which is a game-changer for safe auto-scaling. Here are the best practices for tuning your CN HPA for Iceberg/shared-data workloads. 1. Is Memory 30% too aggressive? Yes, it is far too aggressive for StarRocks. * The "Sticky Memory" Problem: StarRocks CN nodes use a Data Cache (Block Cache) to speed up Iceberg/S3 queries. This cache is designed to "fill up" memory and stay there to serve future requests. It does not release memory back to the OS just because a query finished. * Idle Baseline: Your observation that CNs idle at ~47% is expected behavior. If you set HPA to 30%, the cluster will detect it is "overloaded" (even when idle) and scale out to your 
maxReplicas
immediately, then never scale back down. * Recommendation: * If you use Memory for HPA, set the threshold to 75-80%. * However, for shared-data workloads, Memory is a poor indicator of load. CPU is much more accurate. 2. Metric Strategy: CPU-only, Memory-only, or Both? For Iceberg workloads (which involve heavy Parquet decompression and distributed manifest parsing), CPU is the primary bottleneck and the best metric for HPA. * CPU-only (Recommended): Use a threshold of 60-70%. This ensures that as the "Distributed Planning" and "Scan" tasks increase, the cluster adds more compute power. * Why not both? If you use both, the HPA will scale up if either metric is hit. Since the Data Cache will eventually push memory usage high, the memory metric often triggers "false positive" scale-ups. * Refinement: Use CPU as your scaling trigger and treat Memory limits (
-Xmx
and 
query_mem_limit
) as your per-node protection. 3. Safe Scale-Down and Stabilization To prevent killing CNs mid-query and to avoid "flapping" (scaling up and down rapidly), you should configure the 
behavior
section of your HPA. A. The "Graceful Exit" Mechanism (v4.0.x) In version 4.0.6, when the Operator scales down a CN, it sends a 
SIGTERM
. The CN then: 1. Enters an "Exiting" state. 2. Tells the FE to stop sending it new query fragments. 3. Continues to process existing fragments for a period defined by 
loop_count_wait_fragments_finish
(default 20s). 4. Only shuts down once all fragments are done or the timeout is reached. B. Recommended HPA Behavior Policy To make this production-ready, use a Stabilization Window. This forces the HPA to "wait and see" if the load stays low before removing a node.
Copy code
yaml
autoScalingPolicy:
  maxReplicas: 10
  minReplicas: 3
  hpaPolicy:
    metrics:
      - type: Resource
        resource:
          name: cpu
          target:
            averageUtilization: 70
            type: Utilization
    behavior:
      scaleUp:
        stabilizationWindowSeconds: 0 # Scale up immediately when busy
        policies:
          - type: Percent
            value: 100 # Allow doubling the cluster size if needed
            periodSeconds: 15
      scaleDown:
        stabilizationWindowSeconds: 300 # Wait 5 minutes of low load before scaling down
        policies:
          - type: Pods
            value: 1 # Remove only 1 node at a time
            periodSeconds: 60
4. Critical "Alignment" Checklist To ensure no queries fail during scale-down, you must align three timeouts: 1. StarRocks Graceful Exit: In 
cn.conf
, ensure 
loop_count_wait_fragments_finish
is set (default is 2, which is 20s). For heavy Iceberg queries, you might bump this to 6 (60s). 2. Kubernetes Termination Grace Period: In your 
StarRocksCluster
YAML under 
starRocksCnSpec
, set 
terminationGracePeriodSeconds
to be longer than the StarRocks exit timeout.
Copy code
yaml
    starRocksCnSpec:
      terminationGracePeriodSeconds: 90 # Give the CN 90s to finish queries
3. FE Awareness: Ensure the CNs can talk to the FE during shutdown so the FE knows to stop routing queries to the 
DECOMMISSION
node. Rocky’s Summary Recommendation: * Metric: Use CPU @ 70% as the only scaling metric. * Scale-Down: Set a 
stabilizationWindowSeconds
of 300 (5 minutes). * Graceful Shutdown: Set 
terminationGracePeriodSeconds: 90
in your Pod spec to protect those "hundreds of thousands of files" Iceberg scans that might take a minute to finish. Does this give you a safer path for your CN auto-scaling? References • administration/management/graceful_exit.md • administration/management/resource_management/Query_management.md
Open in Slack
PreviousNext
Powered by