@Rocky I have metadata mem under mem_tracker 19gb and peak is also 19gb why it can be so? it does not free up that space and allocates it statically ?

In StarRocks, it is normal to see the metadata memory usage (often tracked under

table_meta

in the BE

mem_tracker

) reach a high level and stay there, with the current usage matching the peak. This behavior is usually not due to static allocation, but rather an LRU (Least Recently Used) cache that stores metadata to accelerate queries. Why it happens 1. Metadata Cache: The BE stores metadata such as Tablet metadata, Rowset metadata, Column metadata, and indices (

ColumnReader

,

IndexReader

) in memory. Most of this is managed by an LRU cache. 2. Memory Limit Configuration: There is a BE configuration parameter called

metadata_cache_memory_limit_percent

(default is 30%). _ StarRocks calculates a memory limit for this cache based on:

process_mem_limit _ metadata_cache_memory_limit_percent / 100

. * If your BE has a

mem_limit

of 64GB, 30% is approximately 19.2GB. This likely explains why your usage has leveled off at 19GB. 3. Peak vs. Current: Because it is an LRU cache, once the memory is allocated to store metadata, it is held to avoid expensive disk I/O for future queries. The cache will grow until it hits its limit and then stay there, evicting old items only when new ones need space. This is why Current usage often equals Peak usage and does not "free up." Is this a problem? In most cases, no. Keeping metadata in memory is critical for query performance. If the memory were freed, every query would have to reload metadata from disk, significantly increasing latency. How to manage it If you believe 19GB is too much for your specific environment (e.g., you have very few tables but many large queries that need more execution memory), you can adjust the limit: * To reduce the limit: Decrease

metadata_cache_memory_limit_percent

in your

be.conf

(e.g., set it to

10

or

20

). * To check details: You can view a detailed breakdown of what is consuming that memory by visiting the BE's debug web page:

http://<BE_IP>:<BE_HTTP_PORT>/mem_tracker?type=table_meta

Summary: The memory is not "statically allocated" at startup, but it is "statically capped" by a percentage-based cache limit. It remains occupied because StarRocks prefers to keep metadata cached for performance rather than releasing it to the OS. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/resource_management/Memory_management.md • best_practices/primarykey_table.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky and what is saved in metadata and is not 19gb too much?

In StarRocks, seeing 19GB under the metadata mem_tracker is often the result of the system's default configuration rather than a memory leak or static allocation. 1. What is stored in the Metadata Cache? The BE (Backend) metadata cache (tracked as

table_meta

or similar in the mem_tracker) stores structures required to locate and read data. Specifically: * Tablet & Rowset Metadata: Information about data versions, rowset IDs, and which files belong to which tablet. * Segment Metadata: Information about individual data files (segments), such as footers, number of rows, and data boundaries. * Column & Index Readers: Metadata for each column (min/max values, dictionary for strings) and indices (Bloom filters, bitmap indices). * Primary Key Indices: If using Primary Key tables, the index that maps keys to row locations (though this sometimes has its own tracker). 2. Is 19GB "too much"? Whether 19GB is too much depends on your total memory limit and data volume: * The "30% Rule": By default, StarRocks sets

metadata_cache_memory_limit_percent

to 30% of the BE's

mem_limit

. * If your BE has a

mem_limit

of 64GB, then 30% is exactly 19.2GB. * If your usage has hit exactly ~19GB and stayed there, it means your metadata has filled the available cache. This is intentional: StarRocks keeps as much metadata as possible in memory to prevent slow disk I/O during queries. * Data Density: 19GB of metadata suggests a high number of Tablets, Rowsets, or Columns. If you have millions of small files (due to frequent small imports or lack of compaction), metadata usage will balloon. 3. Why it doesn't "free up" This is an LRU (Least Recently Used) cache. * Current = Peak: Once the memory is allocated to store a piece of metadata, it is held until the cache is full. * No Release to OS: When the cache hits its limit (e.g., 19GB), and new metadata needs to be loaded, the system evicts the oldest metadata to make room. It does not return the memory to the Operating System because it assumes it will need that space again immediately for other metadata. 4. How to diagnose and tune If you want to verify if this is "normal" or if you should reduce it: 1. Check the limit: Check your

be.conf

or the BE web UI (

http://<BE_IP>:8040/conf

) for

metadata_cache_memory_limit_percent

. If it's 30, and your

mem_limit

is ~64GB, the behavior is exactly as configured. 2. View details: Go to the BE debug page:

http://<BE_IP>:8040/mem_tracker?type=table_meta

(or just

/mem_tracker

). If a specific table or "ColumnReader" is taking the bulk of the 19GB, you may have too many columns or too many small files. 3. Adjust the limit: If you need more memory for query execution and want to shrink the metadata cache, add this to

be.conf

and restart:

properties
    # Reduce metadata cache to 10% of BE memory
    metadata_cache_memory_limit_percent = 10

4. Check Compaction: If metadata is high because of too many small files, check your compaction scores:

SHOW PROC '/statistic/table_statistic';

If "compaction score" is high, you need to speed up compaction to merge rowsets and reduce metadata overhead. Summary: 19GB is likely the soft cap defined by your configuration. It is not "wasted" memory; it is being used to make your queries faster. If the system needs that memory for a query and the metadata is "evictable," it will be cleared as needed. References • knowledge/trouble_shooting/fe_memory_problems.md • using_starrocks/caching/block_cache.md • faq/Others.md • faq/fe_mem_faq.md