@Rocky I am running starrocks 4.0.10 on K8s in s shared all mode with 2 FEs and 3 CN nodes. I am using HPA configured like so:

hpaPolicy:
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
    - type: Resource
      resource:
        name: memory
        target:
          type: Utilization
          averageUtilization: 70

I noticed, that once the CN nodes are up, the memory fills up quite quicky during queries, and is never released, which causes the HPA to never downscale. I have to restart the nodes, which then clears the memory and only down will the cluster downscale. How can I fix this ?

The behavior you are seeing is expected due to how StarRocks and its underlying memory allocator (TCMalloc) manage memory. Databases like StarRocks are designed to be "memory-hungry"—they treat memory as a resource to be used for caching (Page Cache, Metadata Cache, etc.) rather than immediately returning it to the OS. Why this happens 1. TCMalloc Freelist: When a query finishes, StarRocks releases the memory, but the TCMalloc allocator often keeps it in a "freelist" (its own internal pool) to speed up future allocations. To Kubernetes, this still looks like used memory (RSS). 2. Page/Data Cache: StarRocks (especially in 4.0.x with the unified cache) aggressively caches data to improve query performance. It will only start evicting this cache when memory usage hits the

memory_high_level

(default 80% of

mem_limit

). 3. HPA Metric Mismatch: The Kubernetes HPA uses Pod memory metrics (equivalent to

process_mem

in StarRocks), which includes both actively used memory and cached memory/freelists. Since the cache persists to benefit future queries, the utilization stays above your 70% threshold, preventing a downscale. How to Fix It 1. (Recommended) Switch HPA to CPU-only It is a best practice for database workloads like StarRocks to use CPU utilization as the primary scaling metric. CPU is a "compressible" resource that drops immediately when load stops, making it a much more reliable indicator of when to scale down.

yaml
hpaPolicy:
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

2. Increase Memory Threshold or Target If you must use memory-based HPA, set the threshold higher than the StarRocks cache eviction points. StarRocks' default soft limit is 80%, so an HPA target of 70% will almost always be triggered and never cleared. * Target: Set to 85% or 90% to account for the persistent cache. * StarRocks Config: Lower the

mem_limit

or

storage_page_cache_limit

in your

cn.conf

(via the

StarRocksCluster

CRD) so the process actually frees memory sooner. 3. Adjust TCMalloc Release Behavior You can tune the CN nodes to return memory to the OS more aggressively by modifying

cn.conf

(via

config

in the StarRocks Operator): *

tc_gc_period

: This controls the interval (in seconds) at which TCMalloc attempts to return unused memory to the OS. The default is usually 60s. Reducing this to

10

or

30

can help, though it may slightly increase CPU overhead. *

storage_page_cache_limit

: Limit the Page Cache size (e.g.,

20%

) so it doesn't fill up the Pod's memory allocation. 4. Use HPA

behavior

for Scaling You can configure the HPA to be more aggressive or conservative in its downscaling to allow for temporary memory spikes:

yaml
behavior:
  scaleDown:
    stabilizationWindowSeconds: 300 # Wait 5 mins of low usage before scaling down
    policies:
      - type: Percent
        value: 100
        periodSeconds: 15

Summary Recommendation The most effective fix for StarRocks on K8s is to remove memory from your HPA metrics and rely on CPU. If memory usage is a concern for stability (OOM prevention), rely on StarRocks' internal memory management (

mem_limit

,

query_mem_limit

) rather than K8s autoscaling. References • best_practices/audit_log_resource_group.md • administration/management/resource_management/Memory_management.md • administration/management/monitoring/Monitor_and_Alert.md • knowledge/trouble_shooting/fe_memory_problems.md