@Rocky how it comes that in memory management I see: process memory: 54gb jemmaloc: 54gb query memory: 102gb?

This discrepancy usually indicates a tracking mismatch between StarRocks' internal accounting system (MemTracker) and the actual physical memory allocator (jemalloc). In StarRocks, these metrics represent different things: * Process Memory (54GB): The actual resident memory (RSS) the OS sees or what the allocator reports as "allocated." * Jemalloc (54GB): The total memory currently allocated by the jemalloc allocator for the entire process. * Query Memory (102GB): The sum of memory tracked by StarRocks' manual accounting system (MemTracker) for all active or recent queries. Why Query Memory can be higher than Process/Jemalloc: 1. Memory Tracker Leaks (Most Likely): StarRocks uses manual instrumentation to track memory. When an operator (like a Join or Sort) allocates memory, it calls

consume(size)

on a tracker. If the code forgets to call

release(size)

when that memory is freed, the tracker value will keep climbing even though jemalloc has actually reclaimed the physical memory. 2. Double Counting: In some versions or specific query patterns (e.g., complex shuffles or parallel joins), certain memory blocks might be accounted for by multiple trackers simultaneously (e.g., both an exchange operator and the receiving operator), leading to an inflated "Query Memory" total that doesn't exist in reality. 3. Intermediate Cache/Buffer Tracking: Sometimes, memory used for intermediate results or query-related buffers is "credited" to the Query Pool tracker but might be shared or managed in a way that jemalloc hasn't paged into physical memory yet (though usually, the discrepancy is the other way around in that case). 4. Version-Specific Bugs: There have been known issues in various StarRocks versions where trackers for certain operators (like

SharedScan

or

TableFunction

) were not accurately released. Why this is a problem: StarRocks enforces its memory limits based on the Trackers, not the physical memory. If your

query_pool

tracker hits a limit (e.g., 90% of BE memory), StarRocks will start failing queries with "Memory Limit Exceeded" even if your server has plenty of physical RAM available (since jemalloc only sees 54GB). How to troubleshoot: 1. Check for "ghost" queries: Visit

http://<BE_IP>:<BE_HTTP_PORT>/mem_tracker?type=query_pool

. Look for queries that finished a long time ago but still show high memory consumption. 2. Identify the specific tracker: Driling down into the

/mem_tracker

page will show you exactly which child tracker (e.g.,

HashJoinOperator

,

ExchangeOperator

) is holding the "phantom" 102GB. 3. Release Free Memory: You can try to force jemalloc to release dirty pages, though this won't fix the tracker mismatch:

curl -XPOST http://<BE_IP>:<BE_HTTP_PORT>/memz?release=true

4. Restart the BE: If the trackers have "leaked" significantly, a BE restart is often required to reset the accounting. 5. Upgrade: Check if you are on a version older than v3.1.x or v3.2.x, as many memory tracking improvements and bug fixes were introduced in recent releases. Recommendation: If you see this consistently while no queries are running, it is likely a bug in the version you are using. You should check the StarRocks GitHub issues or community Slack for known memory tracker leaks specific to your version. References • administration/management/resource_management/Memory_management.md • faq/fe_mem_faq.md • faq/Sql_faq.md • administration/management/BE_configuration.md • faq/resource_isolation_faq.md • faq/shared_data_faq.md