@Rocky I keep getting an error while trying to query

ERROR 1064 (HY000): Memory of process exceed limit. read and decompress page Backend: starrocks-share-data-mode-cn-2.starrocks-share-data-mode-cn-search.spinnaker.svc.cluster.local, Used: 6959103736, Limit: 6957847018. Mem usage has exceed the limit of BE: BE:6007098

Yet running free on that host shows

5.9Gi

used and

9.4Gi

available. The cgroups memory usage for that host is hovering around

2Gi

. Why is the error saying there is more memory used than the system or cgroups shows?

The error occurs because StarRocks uses an internal MemTracker system to account for memory usage, which often diverges from the physical memory reported by the OS (

free

) or

cgroups

. Here is why you are seeing a discrepancy and how to address it: 1. Internal Accounting vs. Physical Usage StarRocks does not simply query the OS for its current memory usage when deciding whether to allow a query. Instead, it maintains a tree of internal counters (MemTrackers). * The Mismatch: If StarRocks' internal accounting "leaks" a counter or over-counts an allocation, the Used value in the error message will grow even if the actual physical resident set size (RSS) stays low. * Jemalloc Retention: StarRocks uses

jemalloc

as its memory allocator. Jemalloc often "retains" memory (virtual address space) that it has already requested from the OS but hasn't necessarily paged into physical RAM yet. Recent reports (e.g., StarRocks Issue #67607) indicate that in containerized environments, these internal trackers can significantly over-report usage compared to cgroup stats due to how jemalloc metadata and retained memory are tracked. 2. Why is the limit set to ~6.5GiB? The error shows a limit of

6,957,847,018

bytes (approx. 6.48 GiB). * By default, the StarRocks Compute Node (CN) or Backend (BE) sets its

mem_limit

to 90% of what it detects as the "total memory." * If your host has ~15GiB but the limit is ~6.5GiB, it is likely that: * The

mem_limit

is explicitly set in your

cn.conf

(or

be.conf

). * The process is restricted by a Docker/Kubernetes memory limit (cgroup limit) that StarRocks detected upon startup. * StarRocks is detecting the "available" memory at startup rather than total. 3. Troubleshooting & Resolution To see exactly what StarRocks thinks is consuming that memory, visit the following debug endpoints on the specific node mentioned in the error (

starrocks-share-data-mode-cn-2

): 1. Check MemTrackers:

curl http://<CN_IP>:8040/mem_tracker

Look for the tracker with the highest

consumption

. This will tell you if the memory is tied to the

query_pool

,

load

,

page_cache

, or something else. 2. Check Jemalloc Stats:

curl http://<CN_IP>:8040/memz

Look for "Bytes in page heap freelist" and "Bytes released to OS." If "Bytes in use by application" is low but the internal tracker is high, it confirms a tracking inaccuracy. Recommended Actions * Increase `mem_limit`: Since your host has 9.4Gi available, you can safely increase the memory limit in

cn.conf

to allow the internal tracker more headroom. Set it to a fixed value like

mem_limit = 10G

. * Update Configuration: If you are running in Kubernetes, ensure your Pod resources (requests/limits) match what you expect. StarRocks typically uses the cgroup limit to calculate its 90% default. * Version Note: If you are on an older version, consider upgrading. Recent versions have improved the accuracy of memory tracking in containerized environments to prevent these "false" OOM errors. References • deployment/shared_data/s3.md • deployment/shared_data/gcs.md • deployment/shared_data/azure.md • deployment/shared_data/minio.md • deployment/deploy_manually.md • quick_start/shared-data.md