http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> in SHOW PROC '/global_current_queri...
# questions-and-troubleshooting
z
@Rocky in SHOW PROC '/global_current_queries'; I see that totally running queries use 20gb of ram but in SHOW COMPUTE NODES; I see that memused is 40gb on each node and why it can be? are there some internal processes which can eat ram like vacuum or something?
r
The discrepancy you are seeing is normal. 
SHOW PROC '/current_queries';
only tracks the memory actively used by the execution layer for running queries, while 
SHOW COMPUTE NODES;
(or 
SHOW BACKENDS;
) reports the total memory consumed by the StarRocks process. The "missing" 20GB of RAM is typically consumed by the following internal components: 1. Caches (Major Consumer) StarRocks uses several caches to speed up performance. Even if no queries are running, these caches stay in memory: * Page Cache: Stores data pages from the storage layer to avoid disk I/O. * Metadata Cache: Stores metadata for Rowsets and segments. In newer versions (v3.2.10+), this can use up to 30% of the process memory by default (
metadata_cache_memory_limit_percent
). * Other Caches: Including the Column Cache, Index Cache, and PK (Primary Key) Index (for Primary Key tables, which is often a large consumer). 2. Metadata StarRocks keeps significant metadata in memory for fast access: * Tablet & Rowset Metadata: If you have a large number of tablets, the metadata for schemas, column readers, and index readers can consume several gigabytes. * Primary Key Indexes: If you use the Primary Key model, the entire index for active tablets is held in RAM to support fast updates. 3. Background Tasks (Similar to "Vacuum") While StarRocks doesn't have a "vacuum" command, it performs Compaction in the background: * Compaction: Merges small data files into larger ones. This process requires memory for reading and merging data, which is not tracked as a "user query." * Data Loading: If you have active Routine Loads or Stream Loads, they consume memory via MemTables before flushing data to disk. 4. Process Overhead * TCMalloc/Memory Allocator: The underlying memory allocator (StarRocks uses TCMalloc) often holds onto memory "freelists" to speed up future allocations rather than returning it to the OS immediately. * Thread Stacks & RPC Buffers: General process overhead for managing connections and internal communication. How to see the exact breakdown To see exactly what is eating the memory, you should check the Memory Tracker. You can access it via the web browser for any node:
Copy code
bash
# Replace <IP> and <PORT> with your CN/BE IP and its http_port (default 8040)
http://<BE_IP>:<BE_HTTP_PORT>/mem_tracker
This page provides a detailed hierarchical tree showing how much memory is used by 
query_pool
, 
load
, 
compaction
, 
page_cache
, 
metadata
, etc. Summary of commands: * 
SHOW PROC '/current_queries';
: Memory used by SQL execution. * 
SHOW COMPUTE NODES;
: Total process memory. * 
http://<IP>:<PORT>/mem_tracker
: The ground truth for all memory components. References • administration/management/resource_management/Memory_management.md • administration/management/BE_configuration.md • knowledge/trouble_shooting/fe_memory_problems.md
The discrepancy between 
SHOW PROC '/global_current_queries';
and 
SHOW COMPUTE NODES;
is expected. 
SHOW PROC '/global_current_queries';
only tracks memory used by the query execution layer (the RAM actively allocated for processing your current SQL statements). It does not include long-lived caches or background data management overhead. The "extra" 20GB of RAM you are seeing is likely consumed by the following internal components: 1. Storage Caches (The Largest Consumer) StarRocks keeps data in memory to avoid slow disk I/O. These caches are held even if no queries are running: * Page Cache: Stores frequently accessed data blocks. In many configurations, this can take up a significant percentage of the BE's memory limit. * Metadata Cache: Stores metadata for segments, rowsets, and tablets. In recent versions (v3.x+), metadata caching is more aggressive to support cloud-native and shared-data performance. * Column Cache: Speeds up access to specific data columns. 2. Primary Key Indexes (Table-Specific) If you use the Primary Key model, the primary key index for every active tablet is held entirely in RAM to support high-speed UPSERTS. This can easily consume several gigabytes if you have large tables with many unique keys. 3. Background Tasks (Similar to "Vacuum") While StarRocks doesn't have a command named "vacuum," it runs Compaction continuously: * Compaction: This background process merges data versions and deletes "stale" records (performing the same function as vacuum). It requires memory to read and sort data during the merge. * MemTables (Loading): If you are running Stream Load, Routine Load, or Insert jobs, data is first buffered in "MemTables" before being flushed to disk. 4. Metadata and Allocator Overhead * Tablet/Rowset Metadata: Even "idle" tablets consume a small amount of memory for metadata. With hundreds of thousands of tablets, this adds up. * TCMalloc/Jemalloc Overhead: The underlying memory allocator often keeps memory in "freelists" to speed up future allocations rather than returning it to the OS immediately. How to see the exact breakdown You can see exactly which component is using that 20GB by checking the Memory Tracker on the Compute Node (CN) or Backend (BE). Open your browser and go to:
Copy code
text
http://<NODE_IP>:8040/mem_tracker
_(Replace 
8040
with your 
be_http_port
or 
cn_http_port
if you changed the default.)_ This page will show a hierarchy (e.g., 
query_pool
, 
load
, 
compaction
, 
metadata
, 
page_cache
). If the sum of these doesn't match the process memory, check the 
memz
link on the same page to see the allocator (TCMalloc) overhead. ) statistics. References • administration/management/resource_management/Memory_management.md • using_starrocks/running_queries.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md -PROC) -/) _PROC) References • administration/management/resource_management/Memory_management.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/BE_configuration.md
z
@Rocky yea I see in mem_Tracker page_cache, data_cache and jamalloc_metadata and what are each?
r
The terms you see in the 
mem_tracker
represent different types of caching and internal overhead that explain why the process memory is higher than the active query memory. *1. page_cache* This is an in-memory cache that stores decompressed data pages and index pages. * What it does: When StarRocks reads data from local or external disks, it decompresses the blocks and stores them here. Subsequent queries for the same data can skip the decompression and disk I/O steps. * Why it's large: It uses an LRU (Least Recently Used) strategy and will naturally grow to fill its assigned quota to maximize performance. *2. data_cache (also known as Block Cache)* This is primarily used in shared-data (cloud-native) clusters or for external tables (like HDFS/S3). * What it does: It caches "raw" data blocks from remote storage onto local memory and disks. * Difference from Page Cache: While 
page_cache
stores decompressed pages, 
data_cache
is a more general-purpose block cache used to bridge the latency gap between remote storage (like S3) and the Compute Node. If you are using StarRocks in a cloud-native or shared-data architecture, this is where your "hot" data lives. *3. jemalloc_metadata* This is the internal overhead of the memory allocator (
jemalloc
) itself. * What it does: To manage memory efficiently, 
jemalloc
needs its own RAM to store "metadata" (e.g., tracking which chunks of memory are free, mapping pages, and managing thread-local caches). * Why it's there: In newer versions of StarRocks (v3.2.12+), this is explicitly tracked (
enable_jemalloc_memory_tracker
) to ensure the 
mem_tracker
accurately reflects the total process RAM. It isn't "leaked" memory; it's simply the cost of high-performance memory management. Summary * 
page_cache
 + 
data_cache
: These are "performance" memory—they exist to make your queries faster. * 
jemalloc_metadata
: This is "operational" memory—it's the overhead of the system that allocates memory for everything else. If you want to limit how much RAM these caches use, you can adjust parameters like 
storage_page_cache_limit
or 
block_cache_mem_size
in your BE/CN configuration. References • administration/management/resource_management/Memory_management.md • data_source/data_cache.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md References • administration/management/resource_management/Memory_management.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • administration/management/BE_configuration.md • data_source/data_cache_observe.md
5 Views
Open in Slack
PreviousNext
Powered by