This message was deleted StarRocks #questions-and-troubleshooting

Join Slack

This message was deleted.

# questions-and-troubleshooting

Slackbot

08/03/2026, 2:08 PM

This message was deleted.

Rocky

08/03/2026, 2:08 PM

The error message

Poco::Exception. e.code() = 111, e.displayText() = Connection refused

indicates a network-level failure where the StarRocks Compute Node (CN) attempted to connect to the object storage (S3) endpoint, but the connection was rejected. Error code 111 is the standard Linux error for

ECONNREFUSED

. This typically occurs in Shared-data (Cloud-native) clusters when the

starlet

agent (responsible for object storage interaction) cannot reach the S3 service or its configured proxy. Potential Causes & Troubleshooting 1. Incorrect S3 Endpoint Configuration If you are using a private S3-compatible service (like MinIO or Ceph) or a VPC endpoint, the CN might be trying to connect to an incorrect address or port. * Check
cn.conf
: Look for S3-related configurations or environment variables like

AWS_S3_ENDPOINT

. * Storage Volume: If you created a storage volume, verify the

aws.s3.endpoint

property:

Copy code

sql
    DESC STORAGE VOLUME <volume_name>;

2. Network Connectivity from K8s Pods Since the error is "Connection refused," the CN pod itself cannot reach the target. * Test Connectivity: Exec into the failing CN pod (

kube-starrocks-cn-2

) and try to reach the S3 endpoint:

Copy code

bash
    kubectl exec -it kube-starrocks-cn-2 -- curl -v <your-s3-endpoint>

* DNS Issues: Ensure the pod can resolve the S3 hostname. * Security Groups/Network Policies: Ensure that Egress rules allow traffic from the CN pods to the S3 service on the required port (usually 80, 443, or 9000 for MinIO). 3. S3 Proxy or Gateway Issues If your cluster uses an HTTP proxy to reach AWS S3, the proxy might be down or misconfigured. * Check if

http_proxy

https_proxy

environment variables are set in the CN deployment. * If using an S3 gateway (like a VPC endpoint in AWS), ensure the gateway is properly attached to the subnet where the CNs reside. 4. Starlet Client Timeouts (High Load) In some cases, if the S3 service is overwhelmed, it might refuse connections. You can try increasing the retry logic and timeouts in `cn.conf`:

Copy code

properties
# Increase connection timeout and retries
starlet_fslib_s3client_connect_timeout_ms = 5000
starlet_fslib_s3client_nonread_max_retries = 10

(Note: These parameters require a CN restart to take effect.) Summary Recommendation 1. Verify the S3 endpoint URL and port being used. 2. Run a

curl

test from inside the

kube-starrocks-cn-2

pod to confirm if the endpoint is reachable. 3. If using MinIO/Ceph, ensure the service is healthy and the load balancer is not rejecting connections. References • deployment/shared_data/minio.md • deployment/shared_data/s3.md • administration/Meta_recovery.md • faq/Deploy_faq.md • quick_start/routine-load.md • quick_start/shared-data.md • administration/management/BE_configuration.md

Zurab Kiknavelidze

08/03/2026, 2:18 PM

@Rocky W20260803 130921.157029 131628033885760 fragment_context.cpp:200] [Driver] Canceled, query_id=c02cbf68-8f11-11f1-b7c1-6e82c32d69f1, instance_id=c02cbf68-8f11-11f1-b7c1-6e82c32d6ab0, reason=transmit chunk rpc failed [dest_instance_id=c02cbf68-8f11-11f1-b7c1-6e82c32d6a60] [dest=10.10.10.10:8060] detail:brpc failed, error=Host is down, error_text=[E104]Fail to read from Socket{id=1224 fd=253 addr=10.10.10.10806053178} (0x0x77b5df68a280): Connection reset by peer [R1][E112]Not connected to 10.10.10.10:8060 yet, server_id=1224 [R2][E112]Not connected to 10.210.7.40:8060 yet, server_id=1224 [R3][E112]Not connected to 10.10.10.10:8060 yet, server_id=1224

Rocky

08/03/2026, 2:18 PM

The error message

brpc failed, error=Host is down

combined with

[E104] Connection reset by peer

indicates a networking or process-level failure where the node attempting to send data (the sender) lost its connection to the destination node (

10.10.10.10:8060

). This usually happens when the destination node's process (BE or CN) is no longer available to handle the request. Common Causes & Troubleshooting 1. Destination Process Crash (OOM) The most common cause for a "Connection reset" followed by "Host is down" is that the StarRocks process on the destination node (

10.10.10.10

) crashed or was killed by the operating system. * Check OOM Killer: Run the following on the destination host:

Copy code

bash
    dmesg -T | grep -i oom

* Check Process Status: Check if the BE/CN process is still running or if it has a recent restart time. * Review
be.out
/
cn.out
: Look for stack traces or panic messages in these files on the destination node. 2. Destination Node Logs Examine the

<http://be.INFO|be.INFO>

(or

<http://cn.INFO|cn.INFO>

) logs on the destination node (

10.10.10.10

) around the timestamp

13:09:21

. Look for: * Fatal errors or "Received signal" messages. * High memory usage warnings. * Disconnection events from other nodes. 3. Network Instability If the process is still running, the issue might be network-level (e.g., a firewall, load balancer, or K8s CNI resetting connections). * Check Port 8060: Ensure port 8060 (the bRPC port) is reachable from other nodes:

Copy code

bash
    telnet 10.10.10.10 8060

* Check TCP Backlog: If the server is under heavy load, it might drop connections. Check if

TCPBacklogDrop

is increasing:

Copy code

bash
    netstat -s | grep TCPBacklogDrop

4. bRPC Socket Overload In high-throughput scenarios, if the sender is pushing data faster than the receiver can process, it can hit the

brpc_socket_max_unwritten_bytes

limit. However, this usually results in a

The server is overcrowded

error rather than

Host is down

. * If you see "overcrowded" in other logs, you can try increasing this limit in `be.conf`/`cn.conf`:

Copy code

properties
    brpc_socket_max_unwritten_bytes = 1073741824  # Increase to 1GB or higher

Summary of Next Steps 1. Verify if the BE/CN process at

10.10.10.10

is still alive. 2. If it restarted, check the

dmesg

and

be.out

on that node for Out of Memory (OOM) or Segfaults. 3. Since you previously reported S3 connection issues (

Connection refused

), there is a high likelihood of general network instability within your Kubernetes cluster or environment affecting both node-to-node and node-to-S3 communication. References • faq/operation_maintenance_faq.md • administration/management/BE_configuration.md • faq/Sql_faq.md References • faq/operation_maintenance_faq.md • knowledge/usage/best-practice.md • faq/Deploy_faq.md • faq/Sql_faq.md • faq/loading/Loading_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md

Zurab Kiknavelidze

08/03/2026, 3:05 PM

@Rocky why starrocks compute nodes can go on OOM?

Rocky

08/03/2026, 3:05 PM

StarRocks Compute Nodes (CN) can experience Out-Of-Memory (OOM) issues for several reasons, ranging from query execution patterns to background caching mechanisms in shared-data architectures. 1. Resource Limit Mismatch (K8s vs. StarRocks) This is the most common cause in Kubernetes environments. * The Conflict: By default, StarRocks sets

mem_limit

to 90% of the total physical memory detected. If your K8s Pod has a memory limit (e.g., 32GB) but is running on a node with 128GB of RAM, StarRocks might try to use 90% of 128GB (~115GB), causing the K8s OOM killer to terminate the pod immediately. * Solution: Explicitly set

mem_limit

cn.conf

to match your K8s container requests/limits, or use a percentage that reflects the container's allocation. 2. Data Cache Metadata (Shared-Data Architecture) CNs in shared-data clusters use Data Cache to speed up access to S3/Object Storage. * Metadata Overhead: While the data itself is cached on disk, the index and metadata for that cache are stored in memory. * Known Bug: In StarRocks versions earlier than v3.1.9 or v3.2.4, there was an issue where the

starlet

cache metadata could grow uncontrollably, consuming a massive portion of CN memory. * Identification: Check the CN metrics for

fslib_star_cache_meta_memory_size

. If this is high, it indicates cache metadata pressure. 3. Query Execution & Concurrency * Large Joins/Aggregations: Queries involving massive shuffle operations or large hash tables (e.g., joining two billion-row tables without effective pruning) can exceed the

exec_mem_limit

or the total available memory. * Concurrency: Even if individual queries are small, a high volume of concurrent queries can collectively exhaust the node's memory. * Configuration:

exec_mem_limit

(default 2GB or 0 for auto) controls per-query memory. If set too high, one query can kill the node; if too many queries run at once, they can bypass individual limits but hit the global

mem_limit

. 4. Memory-Intensive Background Tasks * Compaction: Although CNs are stateless, they perform compaction for shared-data tables to merge small files on S3. Each compaction thread (

compaction_memory_limit_per_worker

) defaults to 2GB. Multiple concurrent compaction tasks can lead to OOM. * Loading: Data loading processes (

Stream Load

Broker Load

) consume memory for buffering chunks. This is controlled by

load_process_max_memory_limit_percent

(default 30%). 5. bRPC Buffer Saturation StarRocks uses bRPC for node-to-node data transfers (shuffling). If the network is slow or the destination node is congested, the sender CN will buffer data in memory. If

brpc_socket_max_unwritten_bytes

is reached or many sockets are active, this can contribute to OOM. Troubleshooting Steps 1. Check the OOM Killer: Run

dmesg -T | grep -i oom

on the host to confirm if the OS killed the process. 2. Review
be.out
/ `cn.out`: These files often contain a "Memory Tracker" dump just before a crash, showing which module (Query, Load, Cache, etc.) was using the most memory. 3. Upgrade: If you are on an older v3.x version, upgrade to at least v3.1.9+ or v3.2.4+ to benefit from Data Cache memory optimizations. 4. Adjust `mem_limit`: Ensure

mem_limit

cn.conf

is strictly less than the K8s limit. References • administration/management/BE_configuration.md

Zurab Kiknavelidze

08/03/2026, 3:12 PM

@Rocky WHERE IS MEM_LIMIT for k8s cluster

Rocky

08/03/2026, 3:12 PM

In a Kubernetes cluster, there are two distinct memory limits you must manage to prevent OOM (Out of Memory) kills. If the StarRocks internal limit is higher than the Kubernetes limit, the pod will be killed by the OS/K8s. 1. Kubernetes Container Limit (The "Hard" Limit) This is the limit enforced by Kubernetes. If the pod's memory usage exceeds this, the K8s OOM killer will terminate it immediately. * Operator (CRD): Set this in

starRocksCnSpec

starRocksBeSpec

. * Helm: Set this in the

resources

section of your

values.yaml

Copy code

yaml
# Example for starrocks-operator CRD
starRocksCnSpec:
  requests:
    memory: "64Gi"
  limits:
    memory: "64Gi"  # <--- This is the K8s Limit

────────── 2. StarRocks Internal
mem_limit
(The "Soft" Limit) This is the internal configuration for the StarRocks process. By default, StarRocks sets this to 90% of the physical RAM it detects on the host. Crucial: In K8s, StarRocks might see the host's total RAM (e.g., 256GB) instead of the pod's limit (64GB). You must manually set this to be equal to or slightly less than your K8s limit. Option A: Using the StarRocks Operator You can inject this into

cn.conf

(or

be.conf

) using the

configMapInfo

field in your

StarRocksCluster

YAML: 1. Create a ConfigMap with your desired setting:

Copy code

bash
    kubectl create configmap cn-config --from-literal=cn.conf="mem_limit = 90%"

2. Reference it in your CRD:

Copy code

yaml
    starRocksCnSpec:
      configMapInfo:
        configMapName: cn-config
        resolveKey: cn.conf

Option B: Using the Helm Chart In your

values.yaml

, use the

config

field (or `beConfig`/`cnConfig` depending on the chart version):

Copy code

yaml
starrocks:
  starrocksCnSpec:
    config: |
      mem_limit = 58G  # Set this to ~90% of your K8s limit (e.g., 90% of 64Gi)
    resources:
      limits:
        memory: 64Gi

Recommendation If your K8s limit is 64Gi, set your StarRocks internal

mem_limit

to roughly 58G or 90%. This ensures StarRocks starts its own internal garbage collection and query cancellation before Kubernetes forcibly kills the pod. • Check current internal limit: Run

SHOW BACKENDS\G

SHOW COMPUTE NODES\G

in SQL and look at the

MemLimit

column. • Verify K8s limit: Run

kubectl describe pod <pod_name>

. References • deployment/sr_operator.md • deployment/shared_data/minio.md

Zurab Kiknavelidze

08/03/2026, 3:14 PM

@Rocky so in that case resources: requests: cpu: 28 memory: 120Gi # If you want to remove one resource limit, e.g., cpu, you can set it to cpu: "unlimited". limits: cpu: 28 memory: 120Gi it can go on OOM? I should decrese it?

Rocky

08/03/2026, 3:14 PM

Yes, your current configuration can still lead to an OOM (Out of Memory) kill if the internal StarRocks

mem_limit

is not correctly configured to respect that 120Gi boundary. Here is why this happens and how you should fix it: 1. The "Host RAM" Detection Problem By default, StarRocks sets

mem_limit = 90%

. In many Kubernetes environments, the StarRocks process may detect the total RAM of the physical worker node (the host) rather than the 120Gi limit you set for the Pod. * Example: If your K8s node has 256Gi of RAM but your Pod limit is 120Gi, StarRocks might try to use 90% of 256Gi (~230Gi). * Result: The Pod will exceed its 120Gi limit almost immediately during a heavy query, and Kubernetes will kill it (

OOMKilled

). 2. The Safety Buffer Requirement Even if StarRocks correctly detects the 120Gi limit, 90% (108Gi) might be too aggressive for a container. * Overhead: Memory is also needed for the OS, glibc/tcmalloc overhead, thread stacks, and system caches. * Fragmentation: Memory allocators (like tcmalloc used by StarRocks) don't always return memory to the OS immediately. * K8s Limit: If the process's actual memory usage (RSS) hits exactly 120Gi, Kubernetes kills it instantly. You want StarRocks' internal protection to trigger before Kubernetes steps in. Recommended Action: Use an Absolute Value Instead of relying on the 90% percentage, you should explicitly set an absolute value for

mem_limit

in your

cn.conf

be.conf

that is lower than your K8s limit. Step-by-Step Fix: 1. Keep your K8s limit at 120Gi. 2. Set the internal
mem_limit
to ~108G (90% of 120Gi) or 110G. * In your StarRocks operator YAML or Helm `values.yaml`:

Copy code

yaml
        starRocksCnSpec: # or starRocksBeSpec
          config: |
            mem_limit = 108G
          resources:
            limits:
              memory: 120Gi
            requests:
              memory: 120Gi

3. Verify the change: Run this SQL command in your cluster:

Copy code

sql
    SHOW COMPUTE NODES\G  -- or SHOW BACKENDS\G

Look at the

MemLimit

column. If it shows a value much larger than 120Gi, your node is detecting the host RAM and will OOM. Summary * Do not decrease the 120Gi K8s limit (unless you want to save money). * Do decrease the internal
mem_limit
to a fixed value (e.g.,

108G

) to ensure StarRocks cancels heavy queries before they crash the whole Pod. References • administration/management/BE_configuration.md • deployment/sr_operator.md • administration/management/resource_management/Memory_management.md References • administration/management/resource_management/Memory_management.md • deployment/plan_cluster.md • administration/management/BE_configuration.md • quick_start/helm.md • deployment/sr_operator.md • faq/resource_isolation_faq.md

Zurab Kiknavelidze

08/03/2026, 3:16 PM

@Rocky total ram in container is 125 gb

Rocky

08/03/2026, 3:16 PM

With a container memory limit of 125 GB, your current configuration is at high risk of an OOM (Out of Memory) kill by Kubernetes. Yes, you should decrease the internal StarRocks
mem_limit
setting. You should not necessarily decrease your Kubernetes

limits

(keep those at 125 GB if that's your budget), but you must explicitly tell StarRocks to stay well below that 125 GB mark. Why you must decrease it 1. Host vs. Container Detection: By default, StarRocks calculates its memory limit as 90% of the physical RAM it detects. In many Kubernetes environments, the StarRocks process detects the host node's RAM (e.g., 256 GB or 512 GB) rather than the Pod's 125 GB limit. This causes StarRocks to believe it has ~230 GB+ available, leading it to over-allocate until Kubernetes kills the Pod at exactly 125 GB. 2. Kubernetes Overhead: Kubernetes needs a "buffer" for the container's overhead (OS processes, thread stacks, and memory fragmentation from the

tcmalloc

allocator). If StarRocks uses exactly 100% of the limit, the Pod is killed instantly. Recommended Configuration For a 125 GB container, you should set the internal StarRocks

mem_limit

to roughly 90% of that value, which is about 112 GB. Set this in your
cn.conf
or `be.conf`:

Copy code

properties
mem_limit = 112G

How to apply this in K8s * If using the StarRocks Operator: Add it to the

config

section of your

StarRocksCluster

YAML:

Copy code

yaml
    starRocksCnSpec:
      config: |
        mem_limit = 112G
      resources:
        limits:
          memory: 125Gi
        requests:
          memory: 125Gi

* If using Helm: Update your `values.yaml`:

Copy code

yaml
    starrocksCnSpec:
      config:
        mem_limit: "112G"

How to verify After the Pod restarts, run this SQL command to ensure the limit is applied correctly:

Copy code

sql
SHOW COMPUTE NODES\G  -- or SHOW BACKENDS\G

Check the MemLimit
column. It should show approximately

112.00 GB

. If it shows a much larger number (like

230 GB

), your configuration was not applied, and the node is still at risk of OOM. Summary * K8s Limit: 125 GB (The "hard" wall enforced by Kubernetes). * StarRocks `mem_limit`: 112 GB (The "soft" limit where StarRocks will start rejecting queries to protect itself). ). References • administration/management/resource_management/Memory_management.md • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • best_practices/audit_log_resource_group.md

Zurab Kiknavelidze

08/03/2026, 3:18 PM

@Rocky but in SHOW COMPUTE NODES memlimit I see 102gb how?

Rocky

08/03/2026, 3:18 PM

The reason you see 102GB instead of 125GB (or 112GB) is due to how StarRocks calculates its default memory limit and how it detects memory in a container. There are two likely reasons for this specific number: 1. The 80% "Soft Limit" vs 90% "Hard Limit" StarRocks has two internal memory thresholds: * Hard Limit (90%): The point at which it aggressively fails queries to prevent a crash. * Soft Limit (80%): The point at which it starts garbage collection and backpressure. In many versions and displays (like

SHOW COMPUTE NODES

or older releases), StarRocks may report or default to the 80% threshold. * Calculation: $128\text{ GiB} \times 0.80 = \mathbf{102.4\text{ GiB}}$. * If your underlying K8s node has 128GB of RAM (a standard server size), StarRocks is likely detecting the host's total RAM instead of your container's 125GB limit and applying the 80% soft limit to it. 2. Host Memory Detection (Cgroup vs. Physical) If StarRocks is not correctly reading the Kubernetes cgroup limit (the 120Gi/125Gi you set), it falls back to reading the physical RAM of the worker node. * If your container is running on a node with 128GB of RAM, StarRocks sees 128GB. * $128\text{GB} \times 80\% = 102.4\text{GB}$. Is this a problem? Yes. If

SHOW COMPUTE NODES

shows 102GB, StarRocks will start killing queries once you hit 102GB, even though your container has 125GB available. You are "wasting" about 23GB of your allocated memory. How to fix it You should stop relying on the automatic percentage and set an absolute value to ensure StarRocks uses exactly what you want. In your

cn.conf

(or via the Operator

config

section), set:

Copy code

properties
# Set this to roughly 90% of your 125GB container limit
mem_limit = 112G

Why 112G? * 125GB (Container Limit) is your "Hard Wall." If you touch this, Kubernetes kills the pod. * 112GB (StarRocks Limit) is your "Internal Safety." StarRocks will stop queries at 112GB, leaving ~13GB for the OS, overhead, and memory fragmentation, preventing the whole pod from restarting. To apply this in your K8s YAML:

Copy code

yaml
starRocksCnSpec:
  config: |
    mem_limit = 112G
  resources:
    limits:
      memory: 125Gi
    requests:
      memory: 125Gi

After applying and restarting, run

SHOW COMPUTE NODES\G

again. The

MemLimit

should now show

112.00 GB

. References • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md • faq/resource_isolation_faq.md References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • faq/resource_isolation_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • release_notes/release-2.4.md

Zurab Kiknavelidze

08/03/2026, 3:21 PM

@Rocky no my problem is tht starrocks has gone on OOM via that limits (I have 95% mem_limit) 125gb ram on pod and 102gb I see in SHOW compute nodes

Rocky

08/03/2026, 3:22 PM

The reason you are seeing 102GB in

SHOW COMPUTE NODES

while having a 120Gi limit and 95%

mem_limit

is a common point of confusion in Kubernetes environments. 1. Why do you see 102GB? The math reveals where that number comes from: * 100 GiB is exactly 107.37 GB. * $107.37 \text{ GB} \times 95\% \approx 102 \text{ GB}$. It is highly likely that StarRocks is detecting 100 GiB as your "total memory" instead of your 120Gi limit. This happens because: * Cgroup Detection: StarRocks might be reading the

requests

value (if it was 100Gi) or a parent cgroup limit instead of the Pod's

limits

. * Default Behavior: In some versions, if StarRocks cannot reliably detect the container limit, it may fall back to a smaller value or use a legacy calculation (80% of host RAM). If your host has 128GB, $128 \times 80\% = 102.4 \text{GB}$. 2. Why is it still going OOM? You mentioned you are getting OOM kills even with these limits. This happens because mem_limit = 95%
is far too high. StarRocks only tracks memory used by its internal components (queries, metadata, cache). It cannot track everything. Specifically: * TCMalloc Fragmentation: StarRocks uses

tcmalloc

as its memory allocator. It often holds onto memory even after a query is finished (fragmentation). If StarRocks thinks it is using 100GB, the actual process memory (RSS) seen by Kubernetes could be 130GB+. * System Overhead: Memory for thread stacks, shared libraries, and the operating system is not tracked by the internal

mem_limit

. The OOM Killer Logic: If your internal limit is 102GB but your K8s limit is 120Gi (~129GB), and your fragmentation is higher than 27GB, Kubernetes will kill the Pod before StarRocks even realizes it is out of memory. 3. The Recommended Fix Stop using percentages and set an absolute value for

mem_limit

to create a "safety buffer" between StarRocks and Kubernetes. For a 120Gi (129GB) Pod, try these settings: 1. Set an absolute `mem_limit`: Set it to roughly 85-90% of your limit in GB.

Copy code

properties
    # In cn.conf or be.conf
    mem_limit = 110G

This gives you a ~19GB buffer for tcmalloc fragmentation and OS overhead. 2. Adjust TCMalloc (Optional but recommended): If you see memory not being released, you can make

tcmalloc

more aggressive in returning memory to the system:

Copy code

properties
    # In cn.conf or be.conf
    # Default is usually 5000 (5 seconds)
    tcmalloc_gc_period_s = 2

3. Check the logs: If you still get an OOM, look at the

<http://be.INFO|be.INFO>

log just before the crash. Search for "Memory Tracker Summary". If the "Process" memory is much higher than the "Sum of all trackers," it confirms that fragmentation is the cause. Summary * Decrease your
mem_limit
percentage (to ~85%) or, better yet, use a fixed value like
110G
. * *Do not use 95%*; it leaves no room for the process to breathe, which is why Kubernetes is killing it. References • administration/management/BE_configuration.md • administration/management/resource_management/Memory_management.md • faq/resource_isolation_faq.md • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md • best_practices/audit_log_resource_group.md • quick_start/helm.md • deployment/plan_cluster.md • deployment/sr_operator.md

Rocky

08/03/2026, 3:22 PM

Zurab Kiknavelidze

08/03/2026, 3:27 PM

@Rocky about vacuuming starrocks shared data on object store it can use ram outside this 95%?

Rocky

08/03/2026, 3:28 PM

Yes, vacuuming (Garbage Collection) and Compaction in shared-data clusters can use memory that may not be perfectly accounted for by the internal
mem_limit
trackers, especially in ways that lead to OOM kills when your buffer is too small (like 95%). Here is why your 95% setting is likely failing during vacuum/compaction: 1. The "Untracked" Memory Gap While StarRocks tries to track most memory via its

mem_tracker

, some types of memory are difficult to track or are managed by the system allocator (TCMalloc/Jemalloc): * Metadata Loading: To vacuum data on object storage, StarRocks needs to load Tablet Meta files to compare what is in the metadata vs. what is on S3/OSS. If you have many partitions or tablets, loading this metadata consumes significant RAM. * TCMalloc Fragmentation: This is the biggest culprit. If StarRocks tracks 102GB of usage, the underlying allocator might actually be holding 115GB-120GB from the OS due to "fragmentation" (memory that was used and released by a query but not yet returned to Kubernetes). * RPC & Object Storage SDKs: The S3/Object storage SDKs and network buffers used during vacuuming often use memory outside the core StarRocks "Query Pool." 2. Why 95% is causing your OOM On a 125GB Pod, a 95%

mem_limit

leaves only ~6.25GB for everything else. * Operating System: ~1-2GB. * Thread Stacks: StarRocks is highly multi-threaded; each thread takes memory. * TCMalloc Metadata/Fragmentation: Easily exceeds 10GB on a 125GB system. When a Vacuum or Compaction task starts, it adds a sudden burst of metadata and buffer usage. Because you only have a 6GB "safety zone," Kubernetes kills the Pod (OOM) before StarRocks' internal limit-checking logic even has time to react. How to fix it for Shared-Data If you are running a shared-data cluster on object storage, you need a larger buffer because of the metadata overhead. 1. Reduce
mem_limit
to 85% or an absolute value: Set it to 106G (

125 * 0.85

). This gives you a ~19GB safety buffer for vacuuming, metadata, and fragmentation.

Copy code

properties
    mem_limit = 106G

2. Tune Compaction/Vacuum Threads: If vacuuming/compaction is too aggressive, it can spike memory. Lower the concurrency:

Copy code

sql
    -- In CN/BE config
    compact_threads = 4  -- Reduce if it's currently higher

3. Check Memory Trackers during Vacuum: Visit

http://<CN_IP>:<CN_HTTP_PORT>/mem_tracker

while vacuum is running. Look for "TabletMeta" or "Compaction" trackers. If the "Process" memory is much higher than the sum of trackers, it's fragmentation. Summary: The vacuum process doesn't technically "ignore" the limit, but it uses memory (metadata and system buffers) that often pushes the total process size beyond what Kubernetes allows before the internal StarRocks safety-checks can trigger. Lowering your
mem_limit
to 85% is the standard fix. is the standard fix for this. References • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • knowledge/admin_and_operation/shared_data_garbage_cleanup.md • administration/management/compaction.md

3 Views

Open in Slack

Previous Next