http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> have their been any reported issues...
# questions-and-troubleshooting
b
@Rocky have their been any reported issues around memory leaks in front end nodes
r
Yes, there have been several reported issues and known scenarios regarding memory leaks or excessive memory consumption in StarRocks frontend (FE) nodes. Because the FE is a Java-based process, these often manifest as high JVM heap usage, frequent Full GC, or OOM (Out of Memory) crashes. The StarRocks team has actively addressed these, particularly in versions 3.3.x and later, by introducing new memory tracking and optimization features. 1. Reported and Fixed Bugs Several specific memory leak issues have been identified and patched: * Label Cleaner Leak: A leak was previously reported in the 
label cleaner
(related to load job labels), which caused memory to accumulate in the 
DatabaseTransactionMgr
. This was fixed in versions around v2.5.x and v3.x (Issue #27975). * Load Profile Bloat: In version 3.2.4, an issue was found where enabling load profiles (
enable_load_profile = true
) caused the Leader FE to consume massive amounts of memory when running many stream load tasks. This was fixed in v3.2.x/v3.3.x (Issue #43715). * MemoryUsageTracker overhead: Ironically, the 
MemoryUsageTracker
(introduced to find leaks) was itself reported to cause soaring memory usage in some edge cases (Issue #55646). 2. High Memory Scenarios (Often Mistaken for Leaks) Some "leaks" are actually unbounded growth of specific metadata caches or historical data: * Iceberg Metadata Cache: If you use the Iceberg connector with tens of thousands of tables, the FE may cache metadata without a strict limit, leading to OOM. Users often need to restart or limit the catalog scope (Issue #47630). * Task Run History: The FE stores task run records in memory. If not configured correctly, this can grow large. The parameter 
task_runs_max_history_number
(default 10,000) was introduced to bound this. * Image Checkpoints: Historically, the Leader FE would double its memory usage during metadata checkpointing (saving the 
image
file). * Optimization: From v3.4.0, StarRocks supports offloading image checkpoints to Follower nodes to protect the Leader from OOM. * Optimization: From v3.3.3, image files use "streaming read" to reduce the memory spike during startup. 3. How to Troubleshoot If you are seeing a slow increase in FE memory, StarRocks now provides built-in tools to identify the specific module at fault: * Memory Usage Tracker (v3.3.7+): It regularly logs the memory usage of each internal module (Agent, Catalog, Coordinator, etc.) in the 
fe.log
. * Memory Allocation Profiles (v3.3.6+): You can generate flame graphs of heap allocation under 
fe/log/proc_profile
. * Configuration Fixes: * Ensure 
-Xmx
is set to 70-80% of machine memory. * Set 
MALLOC_ARENA_MAX=1
in 
fe.conf
to prevent glibc memory fragmentation (off-heap memory bloat). Recommendation: If you are experiencing these issues on an older version (e.g., pre-3.2), it is highly recommended to upgrade to at least v3.3.7 or v3.4.x, as these versions include the majority of the fixes and the diagnostic tools mentioned above. References • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • administration/Meta_recovery.md
b
@Rocky my front end nodes memory increases slowly over time and do not drop back down
r
A slow increase in StarRocks Frontend (FE) memory that does not drop back down can be caused by several factors ranging from Java heap management to metadata growth or specific off-heap memory issues. Since the FE is a Java-based process, it is important to distinguish between Heap Memory (managed by JVM) and Off-Heap/Process Memory (managed by the OS). 1. Most Likely Cause: Off-Heap Fragmentation (glibc) If your process memory (RSS) keeps growing but the JVM Heap usage (Old Gen) is stable, the most common cause is glibc memory fragmentation. * The Issue: The default glibc memory allocator creates multiple "arenas" for threads, which can lead to memory being held by the OS even after the FE has "freed" it internally. * The Fix: Add the following environment variable to your 
fe.conf
and restart the FE:
Copy code
bash
    export MALLOC_ARENA_MAX=1
This is a standard recommendation for StarRocks FE nodes to stabilize off-heap memory usage. 2. Metadata & History Growth The FE stores various historical records in memory. If your cluster has high activity (many loads or scheduled tasks), these can grow until they hit their configured limits: * Task Run History: StarRocks stores task execution history. If you have many scheduled tasks, check these parameters: * 
task_runs_max_history_number
(Default: 10,000): Limits the number of records. * 
task_runs_ttl_second
(Default: 7 days): Limits how long they stay in memory. * Stream Load History: * 
stream_load_task_keep_max_num
(Default: 1,000): If you are doing very frequent Stream Loads, the FE keeps metadata for recently completed tasks. * External Catalog Caching: If you use Iceberg, Hudi, or Hive catalogs, the FE caches metadata. If you have a massive number of partitions/files, this cache can grow significantly. 3. Leader FE Specific: Image Checkpointing The Leader FE naturally consumes more memory than Followers because it performs "Checkpoints" (writing the 
image
file). * In older versions, the Leader would essentially double its memory usage during a checkpoint. * Fix (v3.3.3+): "Streaming read" was introduced to reduce this spike. * Fix (v3.4.0+): You can now offload checkpoints to Follower nodes, which significantly reduces the Leader's memory footprint. 4. How to Diagnose the "Slow Leak" StarRocks has built-in tools to help you find exactly which module is holding memory: A. Memory Usage Tracker (v3.2.4+) The FE periodically logs memory usage of internal modules (Agent, Catalog, etc.) into 
fe.log
. Look for lines containing 
MemoryUsageTracker
. You can compare these logs over a few days to see which module's 
current_usage
is increasing. B. Memory Allocation Profiles (v3.3.6+) Check the directory 
fe/log/proc_profile/
. StarRocks automatically generates memory allocation flame graphs (HTML files). * Open these files in a browser to see which Java functions are allocating the most memory. * If you don't see files there, ensure 
proc_profile_mem_enable = true
is set in 
fe.conf
. C. JVM Heap Check Run the following command to see if the memory is actually stuck in the "Old Generation" (O):
Copy code
bash
jstat -gcutil <FE_PID> 1000 10
* If the O column (Old Gen) stays at 90%+ and never drops after a 
FGC
(Full GC), it is a heap-based leak or metadata growth. * If the O column is low but 
top
shows high memory usage, it is an off-heap issue (refer back to 
MALLOC_ARENA_MAX
). Summary Recommendation 1. Immediate: Set 
MALLOC_ARENA_MAX=1
in 
fe.conf
. 2. Verify Version: If you are on a version earlier than v3.2.x, consider upgrading to v3.3.7+, which contains significant memory management improvements and diagnostic tools. 3. Inspect Logs: Check 
fe/log/proc_profile/
for flame graphs to identify the specific module consuming memory. References • faq/fe_mem_faq.md
b
The O column is staying at 90%+ but i am not seeing any FGCs
@Rocky the O column is staying at 90%+ but i'm not seeing any FGCs
r
If you see the O (Old Generation) column staying at 90%+ without the FGC (Full GC) counter increasing, it typically points to one of two things: 1. It’s G1 GC Behavior (Not necessarily a leak) Most modern StarRocks deployments use the G1 Garbage Collector. Unlike older collectors, G1 tries to avoid "Full GCs" (stop-the-world events) at all costs. * Mixed GCs: G1 performs "Mixed GCs" that reclaim parts of the Old Generation concurrently while the application is running. These do not increment the 
FGC
counter in 
jstat
. * Threshold: G1 might allow the Old Gen to stay high (e.g., 90%) if it determines that the cost of reclaiming that memory is higher than the benefit, especially if the "Live" data (actual metadata) is taking up most of that space. 2. The Heap is "Truly Full" of Live Metadata If the memory never drops, it means the JVM thinks that 90% of your heap is Live Data (not garbage). This happens if: * Metadata Growth: Your cluster has grown (more tables, partitions, or tablets) and the 
-Xmx
setting is now too small. * External Catalogs: You are querying large Hive/Iceberg/Glue catalogs. The FE caches this metadata. * Task/Load History: Large numbers of routine loads or scheduled tasks are keeping history in memory. ────────── How to identify the "Memory Eater" Since you are on a version that supports it, you should look for the MemoryUsageTracker output in your 
fe/log/fe.log
. 1. Search your logs for this pattern:
Copy code
bash
grep "MemoryUsageTracker" fe.log | tail -n 20
You will see a breakdown like this: * 
Catalog
: The size of your tables/partitions/tablets metadata. * 
BDBEnvironment
: Memory used by the metadata replication (BDB JE). * 
Query
/ 
Coordinator
: Memory used by active or recently cached queries. * 
Agent
: Memory used for communicating with BE nodes. 2. Check the Flame Graphs (v3.3.6+): Check the directory 
fe/log/proc_profile/
. If you see 
.html
or 
.tgz
files there, open them in a browser. They provide a Flame Graph of exactly what code paths are allocating memory. A very wide bar in the graph indicates a specific module consuming the heap. 3. Manually trigger a Full GC (Test only): To confirm if the memory is "garbage" or "live," you can force a Full GC manually:
Copy code
bash
jcmd <FE_PID> GC.run
* If the O column drops significantly after this, it was just G1 GC being "lazy" with reclamation. * If it stays at 90%, the memory is occupied by live objects, and you likely need to increase 
-Xmx
or check for specific metadata bloat. ────────── Recommended Actions 1. Check `-Xmx`: Ensure your FE heap is at least 8GB-16GB for production clusters, or 70-80% of total RAM. 2. Offload Checkpoints (if on v3.4.0+): If the memory pressure is mainly on the Leader node, set 
enable_checkpoint_offload = true
in 
fe.conf
. This allows Follower nodes to handle the memory-heavy task of saving metadata images. 3. Identify the Module: Use the 
MemoryUsageTracker
log to see if one specific module (like 
Task
or 
Catalog
) is growing daily. Are you seeing this memory pressure on all FE nodes, or only the Leader? References • knowledge/trouble_shooting/fe_memory_problems.md • administration/Meta_recovery.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • administration/management/logs.md
b
@Rocky i don't have MemoryUsageTracker in my logs
r
If you don't see 
MemoryUsageTracker
in your logs, it is likely due to your StarRocks version or your configuration settings. 1. Why it might be missing * Version Requirement: While the internal tracker was introduced in v3.2.4, the feature that automatically prints these summaries to the 
fe.log
 was specifically added in v3.3.7. * Configuration: If you are on v3.2.4+, ensure the feature isn't disabled. You can check this by running:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'memory_tracker_enable';
If it is 
false
, you can enable it dynamically (no restart required):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("memory_tracker_enable" = "true");
2. Alternative: Use "Memory Allocation Profiles" (v3.2.12+) Even without the tracker logs, your FE might be generating Flame Graphs that show exactly which part of the code is allocating memory. * Go to your FE log directory: 
cd fe/log/proc_profile/
* Look for files named 
mem-profile-<timestamp>.html
or 
.tgz
. * Download and open these in a browser. They are much more precise than text logs for finding leaks because they show the exact Java stack traces responsible for the memory growth. 3. Manual Diagnosis (For any version) If you are on an older version or the tools above aren't available, you can use standard Java diagnostic commands: A. The "Lightweight" Check (Object Histogram) Run this to see which types of objects are filling up your 90% Old Gen. It ranks objects by memory consumption:
Copy code
bash
jmap -histo:live <FE_PID> | head -n 30
* What to look for: * `com.starrocks.catalog.Tablet`: Indicates too many tablets/replicas. * `com.starrocks.load.loadv2.LoadJob`: Indicates a leak in load metadata. * `com.sleepycat.je...`: Indicates memory used by the BDB metadata database (standard, but can grow). B. The "Force GC" Test Since you see 90% Old Gen but no 
FGC
, the JVM might just be "lazy" because it thinks it still has 10% headroom. Force a collection to see if the memory is actually garbage or live:
Copy code
bash
jcmd <FE_PID> GC.run
* Watch 
jstat -gcutil
while you do this. If the 
O
(Old Gen) drops significantly, it was not a leak. If it stays at 90%, the memory is held by "Live" metadata objects. 4. Common Culprits for "Live" Memory Growth If the memory is truly "Live" (doesn't drop after manual GC), check these settings which often cause slow growth: * Metadata Checkpoints: If this is the Leader node, it might be the checkpointing process. Consider upgrading to v3.4.0 to use 
enable_checkpoint_offload = true
to move this burden to a Follower node. * Query Dump: Check if 
enable_query_dump
is true. If so, StarRocks keeps query details in memory for debugging, which can add up. * High Partition/Tablet Count: If you have millions of tablets, the FE must keep their locations in memory. You may simply need to increase your 
-Xmx
(heap size) in 
fe.conf
. Recommendation: If you are on a version older than v3.3.7, the best path to stability is upgrading. Many memory management fixes and the very "MemoryUsageTracker" you are looking for were prioritized in the v3.3.x and v3.4.x release cycles. References • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • administration/Meta_recovery.md • administration/management/BE_configuration.md
b
@Rocky this is the top entry in the Object histogram: [Ljava.lang.Object; (java.base@17.0.18)
r
In Java, 
[Ljava.lang.Object;
 is simply an array of Objects (
Object[]
). Seeing this as your top entry in a 
jmap -histo
is common but frustrating, because it's a "container" type. In StarRocks FE, this usually means there is a very large collection (like an 
ArrayList
or 
HashMap
) holding onto metadata. Since the arrays themselves are the contiguous blocks of memory, they show up at the top, even if the "real" data is the thousands of small objects inside them. Here is how to interpret this and find the root cause: 1. What is likely inside those arrays? In StarRocks, a slow-growing 
Object[]
usually points to one of three metadata-heavy areas: * Tablets & Replicas: If you have a large cluster with millions of tablets, the 
Catalog
holds them in massive lists/maps. * Task/Load History: This is the most common cause of "slow leaks." Every Routine Load, Broker Load, and Scheduled Task keeps a history in memory. * BDB JE (Metadata Store): The internal database used for metadata replication (Berkeley DB) uses large object arrays for its internal cache. 2. How to "look inside" the arrays Since 
[Ljava.lang.Object;
is the container, look at the next 5-10 entries in your 
jmap -histo
list. They will reveal what the arrays are actually holding. Common "Inner" Objects to look for: * 
com.starrocks.catalog.Tablet
or `com.starrocks.catalog.Replica`: You have too many tablets for your current Heap size. * 
com.starrocks.load.loadv2.LoadJob
or `RoutineLoadJob`: Your load history is too long. * `com.starrocks.scheduler.TaskRun`: Your scheduled task history is filling up. * 
java.util.HashMap$Node
or `java.util.concurrent.ConcurrentHashMap$Node`: Confirms that the 
Object[]
is likely part of a massive Map. 3. Immediate Actions to Reduce the Growth If you suspect it is metadata/history growth, try adjusting these "TTL" and "Limit" settings in 
fe.conf
(requires restart to take effect for some, but many can be set via 
ADMIN SET FRONTEND CONFIG
): * Reduce Task History:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("task_runs_max_history_number" = "1000");
    ADMIN SET FRONTEND CONFIG ("task_runs_ttl_second" = "86400"); -- 1 day
* Reduce Load History:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "43200"); -- 12 hours (Default is 3 days)
* Metadata Checkpoint Offloading (v3.4.0+): If the memory growth is significantly worse on the Leader node, enable this to move the heavy "image writing" memory spike to a Follower node:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_checkpoint_offload" = "true");
4. The "Final Boss" Diagnostic: Heap Dump If the histogram doesn't make it obvious, you need to see the "Path to GC Root." This tells you exactly which variable in the code is holding onto that giant array. 1. Generate a dump:
Copy code
bash
    jmap -dump:live,format=b,file=fe_dump.hprof <FE_PID>
(Note: This will pause the FE for a few seconds to a minute depending on heap size.) 2. Analyze it: Open the file in Eclipse MAT (Memory Analyzer Tool). 3. Run "Leak Suspects": MAT will explicitly tell you: "The instance of java.util.ArrayList... contains a giant Object[] and is held by [ModuleName]". Which version of StarRocks are you running? Versions prior to v3.2 had fewer protections against history bloat, and upgrading to a recent LTS version (like v3.3.7+) often resolves these "slow increase" issues due to improved metadata handling improvements in-memory history management. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • administration/Meta_recovery.md
b
@Rocky this is the full jmap output root@prod-starrocks-fe-2:/opt/starrocks# jmap -histo:live 18 | head -n 30 num #instances #bytes class name (module) ------------------------------------------------------- 1: 135012851 13018480200 [Ljava.lang.Object; (java.base@17.0.18) 2: 395778509 12664912288 com.google.common.collect.ImmutableMapEntry 3: 55267677 10611393984 com.starrocks.analysis.SlotRef 4: 412713133 9905115192 com.google.common.collect.SingletonImmutableList 5: 166889511 6675580440 com.google.common.collect.ImmutableMapEntry$NonTerminalImmutableMapEntry 6: 4175048 6224386392 [Lcom.google.common.collect.ImmutableMapEntry; 7: 75620238 4234733328 com.starrocks.sql.analyzer.Field 8: 2911984 4193423032 [Ljava.util.Map$Entry; (java.base@17.0.18) 9: 55256566 3465496968 [B (java.base@17.0.18) 10: 75621163 3024846520 java.util.LinkedList (java.base@17.0.18) 11: 120927631 2902263144 com.google.common.collect.RegularImmutableList 12: 82553703 2641718496 java.util.ArrayList (java.base@17.0.18) 13: 55385282 1772329024 java.lang.String (java.base@17.0.18) 14: 7304758 935009024 com.starrocks.catalog.Column 15: 28127397 900076704 com.starrocks.sql.parser.NodePosition 16: 14856773 831979288 java.util.TreeMap$Entry (java.base@17.0.18) 17: 13445429 537817160 java.util.HashMap$Node (java.base@17.0.18) 18: 3318516 465293840 [Ljava.util.HashMap$Node; (java.base@17.0.18) 19: 7021609 449382976 java.util.HashMap (java.base@17.0.18) 20: 2941696 400070656 com.starrocks.analysis.ArithmeticExpr 21: 4929475 394358000 java.util.LinkedHashMap (java.base@17.0.18) 22: 9301835 372073400 org.apache.commons.collections4.map.AbstractHashedMap$HashEntry 23: 10529847 336955104 com.starrocks.sql.ast.QualifiedName 24: 5359455 300129480 com.starrocks.sql.ast.SelectListItem 25: 7304840 292193600 com.starrocks.catalog.ColumnStats 26: 1764306 282288960 com.starrocks.analysis.FunctionCallExpr 27: 1726465 262422680 com.starrocks.analysis.BinaryPredicate 28: 60702 248270176 [Lorg.apache.commons.collections4.map.AbstractHashedMap$HashEntry;
r
The 
jmap
output you provided is very revealing. Your FE is currently holding massive amounts of SQL metadata in its live heap. The top entries—`SlotRef` (55 million), 
Field
(75 million), 
ImmutableMapEntry
(395 million), and 
SingletonImmutableList
(412 million)—are the building blocks of analyzed SQL expressions. Since they appear in a 
jmap -histo:live
(which forces a Full GC before counting), they are "Live," meaning they are reachable from a GC root and cannot be collected. The Diagnostic Breakdown 1. Scale of Metadata: You have 7.3 million 
Column
and 
ColumnStats
objects. This suggests a very large schema (e.g., ~70,000 tables if they have 100 columns each, or a massive number of partitions/columns in external catalogs like Hive/Iceberg). 2. Expression Bloat: 55 million 
SlotRef
objects means each of your 7.3M columns is being referenced or analyzed ~8 times across your long-lived metadata. 3. Container Overhead: The 395 million 
ImmutableMapEntry
and 412 million 
SingletonImmutableList
objects are Guava collections holding those expressions and their properties. At ~12GB and ~10GB respectively, these containers are consuming more memory than the data they hold. Potential Root Causes 1. Materialized View (MV) & View Metadata This is the most common culprit for millions of 
SlotRef
objects. StarRocks stores the analyzed statement of every View and Asynchronous Materialized View in memory. * If you have thousands of complex Views or MVs, especially ones that reference many columns or have many partitions, the FE keeps the full analyzed expression tree (
SlotRef
, 
Field
, 
Expr
) for every single one of them to facilitate query rewriting and planning. * Action: Check your MV and View count: 
SHOW MATERIALIZED VIEWS
and 
SHOW PROC '/views'
. Drop any unused or redundant MVs/Views. 2. External Catalog Caching (Hive/Iceberg/Paimon) If you are using external catalogs, StarRocks caches metadata to speed up planning. * There were known issues in older v3.x versions where metadata for external tables was not effectively bounded or where 
ImmutableMap
objects were being duplicated excessively. * Action: If you use Iceberg/Hive, consider upgrading to v3.3.7+ or v3.4.x. Many optimizations were added specifically to reduce the footprint of 
SlotRef
and 
Field
in external table planning. 3. Information Schema Cache StarRocks caches 
information_schema
data. If you have 7.3 million columns, the internal representation of these system tables becomes a giant memory consumer. * Action: Check if your memory usage spikes when BI tools (like Tableau or Superset) are scanning your schema. 4. Query Dump Feature If you have 
enable_query_dump = true
in your 
fe.conf
, the FE saves the serialized plan and expressions for recent queries to help with debugging. * Action: Ensure this is disabled:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_query_dump';
Recommended Steps to Reclaim Memory 1. Analyze MV History: If you use Asynchronous MVs, the task history can sometimes hold onto old references.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("task_runs_max_history_number" = "500");
    ADMIN SET FRONTEND CONFIG ("task_runs_ttl_second" = "86400");
2. Check Routine Load Bloat: While not shown in the top 10, Routine Load history can bloat the heap over time.
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("label_keep_max_second" = "43200"); -- 12 hours
3. Identify the "Owner" of SlotRefs: If you are comfortable with it, the only way to be 100% sure which module "owns" those 55 million 
SlotRef
objects is a Heap Dump:
Copy code
bash
    jmap -dump:live,format=b,file=fe_leak.hprof <PID>
Open it in Eclipse MAT and look at the "Path to GC Root" for a 
SlotRef
object. It will likely lead to a 
MaterializedView
or 
View
object in the 
GlobalStateMgr
.
Summary: Your heap isn't "leaking" in the traditional sense; it is full of legitimate metadata. If you cannot reduce the number of columns, views, or MVs, you simply need to increase your FE 
-Xmx
. Based on your histogram, your metadata alone is taking up ~50GB+. If your heap is 64GB, 90% usage is expected. You likely need a 128GB heap for this scale of metadata. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_deadlock_problems.md
b
@Rocky how do i get the count of all partitions in my cluster
r
To get the total count of partitions across your entire cluster, the easiest way (available in v3.1 and later) is to query the system metadata views. 1. Using 
information_schema
(Recommended for v3.1+) Run the following SQL to get a global count of all partitions in the internal (default) catalog:
Copy code
sql
SELECT count(*) FROM information_schema.partitions_meta;
If you want to see which databases or tables have the most partitions (to help diagnose your memory issue), you can run:
Copy code
sql
SELECT 
    DB_NAME, 
    TABLE_NAME, 
    COUNT(*) as partition_count 
FROM information_schema.partitions_meta 
GROUP BY DB_NAME, TABLE_NAME 
ORDER BY partition_count DESC 
LIMIT 20;
2. Using 
SHOW PROC
(For all versions) If you are on an older version or want to see the hierarchy, you can use the 
PROC
(procedure) system. This is a virtual file system for StarRocks metadata. 1. Get a list of Databases: 
SHOW PROC '/dbs';
2. Drill into a Database: 
SHOW PROC '/dbs/<db_id>';
(This lists tables and their partition counts in a column). 3. FE Metrics (Quickest Overview) If you have many millions of partitions, the SQL query above might take a few seconds. You can check the current live count of "registered" partitions via the FE metrics:
Copy code
sql
-- This table was added in v3.1 to expose Prometheus-style metrics via SQL
SELECT * FROM information_schema.fe_metrics WHERE NAME LIKE '%partition%';
Look for 
starrocks_fe_partition_count
(if available in your specific version). Why this matters for your memory issue: In your previous 
jmap
output, you had 7.3 million 
Column
objects but 
Partition
objects didn't even make the top 30. This suggests: * Your "width" is the problem, not necessarily your "depth": You have an enormous number of columns/tables (schema bloat). * Analyzed Expression Bloat: The 55 million 
SlotRef
and 75 million 
Field
objects are the analyzed representations of your SQL structures (Views, MVs, or cached queries). If your partition count is also in the millions, it compounds the problem because StarRocks must keep the metadata for every column per partition in certain internal caches. If 
partitions_meta
returns a number > 100,000, you should definitely consider increasing your FE Heap (
-Xmx
) to at least 64GB or 128GB to handle that volume of metadata. References • sql-reference/information_schema/partitions_meta.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md References • sql-reference/information_schema/partitions.md • sql-reference/sql-statements/table_bucket_part_index/SHOW_PARTITIONS.md • sql-reference/information_schema/partitions_meta.md
b
@Rocky what is the recommended maximum number of partitions and what if someone needs more than that
r
In StarRocks, the "maximum" number of partitions is more of a performance and memory guideline than a hard code limit. Because every partition is a catalog object that the Front End (FE) must track in its JVM heap, exceeding recommended counts leads to the exact high memory usage and 
SlotRef
bloat you are currently seeing. 1. The Recommended Limits * Per Table (Practical Limit): 4,096 partitions. _ _Why?* This is the default limit for many internal batch operations (
max_partitions_in_one_batch
). While you can technically go higher, performance for metadata operations (like 
SHOW PARTITIONS
or schema changes) starts to degrade. * Per Table (Red Flag Limit): 100,000 partitions. _ _Why?* At this scale, the FE memory footprint becomes significant. It also puts extreme pressure on the BE (Backend) compaction process, as each partition maintains its own set of versioned files. * Cluster-Wide: There is no fixed number, but it is limited by your FE Heap Size. For every 1 million tablets (Partitions $\times$ Buckets $\times$ Replicas), you typically need at least 16GB–32GB of FE Heap. ────────── 2. What if you need more than that? If your business logic requires high-cardinality partitioning (e.g., partitioning by 
tenant_id
and 
day
), use these strategies to stay performant: A. Use Mixed Granularity (StarRocks v3.4+) This is the most powerful feature for reducing partition count. It allows you to keep hot data at a fine granularity (e.g., Daily) and cold data at a coarse granularity (e.g., Monthly). * How: You can merge old daily partitions into a single monthly partition using 
ALTER TABLE ... MERGE PARTITIONS
. * Benefit: You get precise pruning for recent queries while cutting your total metadata by ~30x for historical data. B. Coarsen the Partition Key If you have 10 years of daily data, that's 3,650 partitions. If you also partition by 
region_id
(e.g., 50 regions), you suddenly have 182,500 partitions. * Alternative: Partition by 
Month
instead of 
Day
. Use Bucketing (the second level of distribution) for the high-cardinality ID. * Rule of thumb: A partition should ideally hold 10GB to 100GB of data. If your daily partitions are only a few hundred MBs, your granularity is too fine. C. Use "Dynamic Partitioning" with TTL Automatically drop old partitions to keep the total count stable.
Copy code
sql
PROPERTIES (
    "dynamic_partition.enable" = "true",
    "dynamic_partition.time_unit" = "DAY",
    "dynamic_partition.start" = "-30", -- Keep 30 days of history
    "dynamic_partition.end" = "3"      -- Create 3 days in advance
);
D. Optimize FE for Metadata Heavy Clusters (v3.3/v3.4+) If you must have millions of partitions/tablets, enable these settings in 
fe.conf
to protect the Leader node: * 
enable_checkpoint_offload = true
(v3.4.0+): Moves the heavy memory-intensive "metadata image" creation to a Follower FE, preventing the Leader from OOM-ing during checkpoints. * Increase Heap Size: Set 
-Xmx
to 64G or 128G if your 
jmap
continues to show millions of 
SlotRef
and 
Field
objects. E. External Tables For extremely cold historical data that you rarely query but need to keep, consider exporting it to an External Catalog (like Iceberg or Hive on S3/HDFS). StarRocks can still query it, but it won't have to manage the granular tablet/replica metadata in the same way as internal tables. Summary Advice: If you are hitting the limits, check your tablet count (Partitions $\times$ Buckets). If you have many partitions with only a few MBs of data, you should reduce the bucket count or coarsen the partition time granularity to merge small files and metadata. References • knowledge/trouble_shooting/fe_memory_problems.md • knowledge/admin_and_operation/shared_data_compaction.md
b
@Rocky how would i know if there was a memory leak by looking at the jmap output?
r
Identifying a memory leak from a 
jmap -histo:live
output requires looking for persistence, unbounded growth, and contextual mismatch. In your specific output, the presence of 55 million 
SlotRef
 and 75 million 
Field
 objects is the most critical signal. Here is how to determine if this is a leak or just a very large (but legitimate) workload: 1. The "Live" Requirement The 
-histo:live
flag is crucial. It forces a Full GC before taking the snapshot. * If it's NOT a leak: The numbers should drop significantly or stabilize after the 
jmap
command finishes. * If it IS a leak: The numbers will remain extremely high (like your 55M 
SlotRef
) because something in the StarRocks code is still holding a "strong reference" to them, preventing the Garbage Collector from ever deleting them. 2. Contextual Mismatch (The "Smell Test") Ask yourself if the scale of the objects matches your actual cluster schema: * Schema Size: You have 7.3M 
Column
objects. If you have 70,000 tables with 100 columns each, this is "legitimate" heavy state. However, if you only have 1,000 tables, then 7.3M 
Column
objects is a definite leak (likely related to metadata versions or failed schema changes not being cleaned up). * Expression Ratio: A 
SlotRef
is an analyzed reference to a column in a SQL statement. Having 55 million of them for 7 million columns (~8 references per column) is common if you have Materialized Views or heavy View nesting. If you do NOT use many MVs or Views, 55M 
SlotRef
objects is a metadata leak. 3. StarRocks-Specific Leak Patterns Looking at your top entries, here are the likely "leaky" culprits in StarRocks FE: * Analysis Cache Leak: StarRocks caches the analyzed results of SQL expressions to speed up planning. In some older v2.5 and early v3.x versions, these caches (containing 
SlotRef
and 
Field
) could grow without bounds if certain complex queries were run repeatedly. * Materialized View Metadata: If you have created and dropped many MVs, or have MVs with thousands of partitions, the internal 
MvRewriteContext
or 
AnalysisContext
can sometimes hold onto old 
SlotRef
objects. * Query Dump / Profile Bloat: If 
enable_query_dump
is on, or if 
query_detail_limit
is set very high, the FE stores the full analyzed AST (Abstract Syntax Tree) for every query. This is a common cause for millions of 
ImmutableMapEntry
and 
SlotRef
objects. 4. How to Confirm the Leak (The "Multiple Snapshot" Method) To prove it's a leak, you need a timeline. Run the following command three times, spaced 1 hour apart, during a period of normal or low activity:
Copy code
bash
# Capture the top 20 StarRocks-specific classes
jmap -histo:live <pid> | grep -E "com.starrocks|java.util|google.common" | head -n 20 > jmap_1.txt
Compare the results: 1. Stable: If 
SlotRef
stays at 55M across all three snapshots, it is Heavy State (StarRocks just needs that much memory for your current Views/Metadata). 2. Growing: If it goes from 55M -> 58M -> 62M while your number of tables/views hasn't changed, it is a Memory Leak. Recommendation Given you have 395 million 
ImmutableMapEntry
 objects (Guava maps), this is often a sign of Metadata analysis bloat. * Check your version: If you are below v3.2.10 or v3.3.3, there were several fixes for 
SlotRef
and 
Field
leaks. * Check MV Count: 
SELECT count(*) FROM information_schema.materialized_views;
. If this is high (>500), the memory usage is likely "by design" and you need to increase your FE 
-Xmx
. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md
b
@Rocky we only have 117 tables and 13 materialized views
r
With only 117 tables and 13 materialized views, seeing 55 million 
SlotRef
 and 7 million 
Column
 objects is a definitive sign of a massive memory leak or a severe metadata bug. In a healthy StarRocks cluster of this size, you would typically see fewer than 100,000 such objects. Here is a breakdown of what is likely happening and how to fix it. 1. The "Column" Count: Shard/Tablet Metadata Leak The 7.3 million 
Column
objects are mathematically impossible for 117 tables (it would imply 62,000 columns per table). This usually points to a Shard/Tablet Metadata Leak: * The Bug: There is a known critical bug in StarRocks (fixed in v3.2.10+, v3.3.3+, and v4.0.5) where metadata for "shards" (the internal representation of tablets) is not correctly deleted from the FE memory even after data is compacted or partitions are dropped. * The Symptom: Objects like 
Column
, 
Tablet
, and 
Replica
accumulate indefinitely. * Check: Run 
SHOW PROC '/statistic';
and compare the "TabletCount" to your actual expected tablets. If the number in memory is drastically higher than your data volume suggests, you are hitting this leak. 2. The "SlotRef/Field" Count: Expression Cache & MV Bloat 
SlotRef
and 
Field
are part of the analyzed SQL expression tree. 55 million of these suggest your FE is holding onto millions of Query Plans. * Query Profile Storage: Check your 
fe.conf
for 
query_detail_limit
. If this is set very high (e.g., >10,000), StarRocks stores the full AST (including all `SlotRef`s) for every query in memory. * Materialized View Nesting: If your 13 MVs are complex or deeply nested, the MV rewrite engine may be generating millions of temporary expression objects that are not being garbage collected. * Internal Statistic Cache: StarRocks caches column statistics. If you have a high churn of partitions, the 
statistic_cache_columns
can sometimes lead to expression bloat. 3. Immediate Action Plan Step A: Upgrade (Highly Recommended) If you are on a version earlier than v3.2.10 or v3.3.3, your issue is almost certainly caused by the fixed bugs mentioned above. Upgrading is the only permanent solution for the 
Shard
and 
SlotRef
leaks. Step B: Configuration Workaround If you cannot upgrade immediately, try to flush the caches: 1. Clear Query Detail: 
ADMIN SET FRONTEND CONFIG ("query_detail_limit" = "100");
(Reduces the number of query profiles kept in memory). 2. Disable Query Dump: 
ADMIN SET FRONTEND CONFIG ("enable_query_dump" = "false");
(Stops the FE from saving query plan structures to disk/memory). 3. Check for "Zombie" Tasks: Run 
SHOW ROUTINE LOAD
and 
SHOW ALTER TABLE
. If you have thousands of "PAUSED" or "CANCELLED" jobs, they may be holding onto metadata. Clean them up using 
STOP ROUTINE LOAD
or 
DROP
commands. Step C: Enable Metadata Offloading If you are on v3.4.0+, ensure the Leader FE isn't doing the heavy lifting for checkpoints:
Copy code
bash
# Add to fe.conf and restart
enable_checkpoint_offload = true
This moves the memory-heavy task of saving metadata to a Follower FE, preventing the Leader from crashing during the "Save Image" process. Summary Your object counts are ~1,000x higher than they should be for your schema size. This is not caused by "too much data," but by the FE failing to release metadata for old query plans and deleted tablet versions of your tables. You should treat this as a high-priority bug and plan an upgrade to the latest patch version of your current release branch. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • knowledge/trouble_shooting/fe_deadlock_problems.md
b
@Rocky i am on version 4.0.9
r
Since you are on version 4.0.9, seeing 55 million 
SlotRef
 and 7.3 million 
Column
 objects with only 117 tables is a massive anomaly. Even with 117 tables having 100 columns each, you should only have ~11,700 
Column
objects. Having 7.3 million suggests the Front End (FE) is effectively keeping 600+ historical versions of your entire catalog in memory simultaneously, or there is a specific internal cache that is not evicting. Here is the diagnosis and the specific steps you should take for version 4.0.x: 1. The Likely Culprits in v4.0.x Based on your object counts, the issue is likely one of the following: * Query Profile & AST Bloat (Most Likely for 
SlotRef
): StarRocks 4.0 has enhanced query profiling. If 
enable_collect_query_detail_info
is 
true
or 
query_detail_limit
is set high, the FE stores the full analyzed Abstract Syntax Tree (AST) for thousands of queries. Each AST contains hundreds of 
SlotRef
and 
Field
objects. * Action: Check if 
query_detail_limit
is set to a large value (default is typically 100). If you run high-concurrency short queries, this can eat GigaBytes of RAM. * Metadata Version/Snapshot Leak: There is a known pattern where the FE fails to clean up old 
Column
objects after metadata changes (like 
ALTER
or partition drops). The 7.3 million 
Column
objects strongly point to a "Catalog Version Leak" where old versions of the table schema are being pinned in memory by an unclosed reference. * Materialized View Refresh Context: Even with only 13 MVs, if they refresh frequently, version 4.0.9 has a specific fix (#71265) for the MV scheduler continuing to run even after MVs become inactive. Check if your MV refresh logs show excessive "Analyze" or "Rewrite" activity. 2. How to Verify the Leak (Commands) Run these commands in your MySQL client to see where the metadata "zombies" are hiding: * Check the Tablet/Column Ratio:
Copy code
sql
    SHOW PROC '/statistic';
Look at the TabletCount. If your 117 tables suggest you should have 10,000 tablets but this shows 1,000,000+, you have a metadata leak. * Check for Query Profile Memory:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'query_detail_limit';
If this is > 100, try reducing it: 
ADMIN SET FRONTEND CONFIG ("query_detail_limit" = "10");
. * Check for "Zombie" Tasks: Check 
SHOW ROUTINE LOAD
and 
SHOW ALTER TABLE
. If there are thousands of 
FINISHED
or 
CANCELLED
tasks, the FE might be holding their column metadata. 3. Immediate Mitigation for v4.0.9 1. Flush Query Detail Info: If you suspect query profiles are the cause:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("query_detail_limit" = "0");
    -- Wait 1 minute for GC to kick in, then set it back to 10-20.
2. Enable the Memory Usage Tracker (New in v3.3.7 / 4.0): Check your 
fe.log
for entries starting with 
(MemoryUsageTracker|...)
. StarRocks 4.0.x will periodically print which module is consuming the most memory. Search for "Module Metadata" or "Module Query" in your logs to see which one has the multi-million object counts. 3. Check Catalog Recycle Bin: If you recently dropped many tables or partitions, they stay in the recycle bin for 24 hours (default).
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "3600"); -- Reduce to 1 hour
4. Is it a bug? Given you are on 4.0.9, you have the latest stable patches. However, if your 
jmap
still shows these counts after a 
live
GC and after reducing 
query_detail_limit
, it is a regression bug specifically related to metadata versioning. Recommendation:
If the object counts don't drop after flushing the 
query_detail_limit
, you should restart the Leader FE. Because StarRocks metadata is persisted to an image, a restart will "prune" the in-memory leaks and only load the current valid catalog. If the memory creeps back up within hours, it confirms an active leak in your query or query planning/caching logic. References • knowledge/trouble_shooting/fe_memory_problems.md • faq/fe_mem_faq.md • administration/management/FE_configuration.md • faq/Others.md • knowledge/trouble_shooting/fe_deadlock_problems.md
b
@Rocky 
ADMIN SET FRONTEND CONFIG ("query_detail_limit" = "100");
reports that query_detail_limit does not exist
r
In StarRocks v4.0.x, the configuration parameter names and the internal memory management for query details have been updated. If 
query_detail_limit
is reporting as non-existent, it is because that specific parameter has likely been renamed or moved in the v4.0 architecture. 1. Finding the correct parameter In version 4.0.9, use the following command to find the exact parameter name on your system:
Copy code
sql
ADMIN SHOW FRONTEND CONFIG LIKE '%query%detail%';
Commonly, the parameter you are looking for is now: * 
max_query_detail_info_num
: This controls the maximum number of query detail records (which include the heavy 
SlotRef
and 
Field
objects) kept in the FE memory. * 
query_info_result_limit
: In some v4 sub-versions, this is used to cap the query info result queue. Action: If you find 
max_query_detail_info_num
(or a similar name), set it to a very low value (e.g., 
10
) to see if the 
SlotRef
count in 
jmap
drops:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("max_query_detail_info_num" = "10");
2. Diagnosis: Why 55 million 
SlotRef
? A 
SlotRef
represents a column reference in a SQL expression. Having 55 million of them for only 117 tables is a "Metadata Plan Leak." In v4.0.9, this usually happens for one of two reasons: * Persistent Query Profiles: If 
enable_collect_query_detail_info
is 
TRUE
, StarRocks stores the full plan tree for every query. If you have high-frequency queries, this queue fills up with millions of objects. * Fix: 
ADMIN SET FRONTEND CONFIG ("enable_collect_query_detail_info" = "false");
* Materialized View Refresh Context: Even with only 13 MVs, if they refresh very frequently or have complex SQL, the FE might be failing to release the "Analyzer" state after the refresh job completes. 3. Use the StarRocks Memory Tracker (v4.0 feature) Instead of guessing via 
jmap
, StarRocks 4.0 has a built-in Memory Usage Tracker that logs exactly which module is at fault every minute. 1. Open your 
fe.log
on the Leader FE. 2. Search for the string 
MemoryUsageTracker
. 3. You will see a breakdown like this:
Copy code
text
    Module Metadata - Catalog estimated 45GB of memory. Contains Column with 7300000 object(s).
    Module Query - QueryDetailQueue estimated 20GB of memory. Contains SlotRef with 55000000 object(s).
* If "Module Query" is high: The issue is query profile/detail caching (fix via config above). * If "Module Metadata" is high: You have a Metadata Version Leak. Even if you have only 117 tables, StarRocks might be keeping thousands of historical "versions" of those tables in memory due to a bug in the 
CatalogRecycleBin
or 
StarMgr
. 4. Immediate Mitigation If the memory does not drop after changing the configs, perform a Follower-to-Leader Switch or a FE Restart: 1. StarRocks FE metadata is periodically saved to an "Image" file. 2. A restart forces the FE to reload only the current valid metadata from the image, effectively wiping out any "zombie" objects like your 7.3 million 
Column
entries that shouldn't exist. 3. If the memory climbs back up to millions of objects within hours, it confirms a leak in your specific query pattern or MV refresh logic. Summary of recommended settings for v4.0.9 to reduce FE pressure:
Copy code
sql
ADMIN SET FRONTEND CONFIG ("enable_collect_query_detail_info" = "false");
ADMIN SET FRONTEND CONFIG ("query_detail_explain_level" = "NORMAL");
ADMIN SET FRONTEND CONFIG ("catalog_trash_expire_second" = "3600"); -- Clean up dropped items faster
References • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/FE_configuration.md • faq/fe_mem_faq.md
b
@Rocky how can i see which FE variables different from the defaults
r
To see which StarRocks configuration variables or parameters are different from their defaults, you should look at two different areas: FE/BE Configuration Parameters (those set in `fe.conf`/`be.conf`) and System Variables (those set via 
SET GLOBAL
). 1. BE Configuration (Different from Defaults) The most direct way to see which BE configurations have been modified is by querying the 
information_schema
. This table explicitly includes a 
DEFAULT
column, making it easy to filter for changes.
Copy code
sql
SELECT 
    BE_ID, 
    NAME, 
    VALUE as current_value, 
    `DEFAULT` as default_value 
FROM information_schema.be_configs 
WHERE VALUE != `DEFAULT`;
2. FE Configuration (Parameters in 
fe.conf
) For FE configuration, the SQL interface is slightly different. The 
ADMIN SHOW FRONTEND CONFIG
command does not currently include a "DefaultValue" column in the standard output. * To view current settings:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG;
* To identify changes: Since there is no "diff" flag in SQL for FE, the best way is to check your 
fe.conf
file on the FE host. Any parameter listed in that file is an override of the internal default. _ _Note:* If you used 
ADMIN SET FRONTEND CONFIG
to change a variable dynamically, it will appear in 
ADMIN SHOW FRONTEND CONFIG
but will revert upon restart unless added to 
fe.conf
. 3. System Variables (Session & Global) System variables (like 
exec_mem_limit
, 
parallel_fragment_exec_instance_num
, etc.) are viewed via 
SHOW VARIABLES
. * Global Variables: To see variables that have been changed at a system-wide level:
Copy code
sql
    SELECT * FROM information_schema.global_variables;
* Session Variables: To see what is active in your current connection:
Copy code
sql
    SELECT * FROM information_schema.session_variables;
Summary Table Scope | Command / Table | Has 
DEFAULT
column? BE Config | 
information_schema.be_configs
| Yes (Use 
VALUE != DEFAULT
) FE Config | 
ADMIN SHOW FRONTEND CONFIG
| No System Vars | 
information_schema.global_variables
| No Pro Tip: If you are trying to debug the memory issue mentioned earlier, check the 
IsMutable
column in 
ADMIN SHOW FRONTEND CONFIG
. If a variable is mutable, it might have been changed at runtime. If you suspect a leak, comparing 
ADMIN SHOW FRONTEND CONFIG
against a fresh installation's defaults or the official documentation is the standard procedure. References • sql-reference/information_schema/be_configs.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/config_vars/ADMIN_SHOW_CONFIG.md • sql-reference/information_schema/fe_metrics.md • sql-reference/information_schema/fe_threads.md
b
@Rocky some of my FE settings differ and i have not changed them. for example ``enable_query_queue_v2`` is set to false, but the docs say that its default is true
r
This is a common point of confusion when looking at the StarRocks documentation, which often reflects the "Latest" or "v4.1+" state by default. For the specific variable you mentioned: 1. 
enable_query_queue_v2
Default You are correct that the current documentation for "Latest" or "v4.1" says the default is 
true
. However, for your version (v4.0.9), the default is actually 
false
. * v4.0.x and below: The default is 
false
. * v4.1.x and above: The default was changed to 
true
to enable the more advanced resource-group-aware queuing by default. 2. Why do other settings differ? There are three main reasons why your settings might differ from what you see in the online documentation: * Version Mismatch in Docs: Most users land on the "Latest" version of the StarRocks docs via Google. Always check the version dropdown in the top-right of the StarRocks documentation site and set it to v4.0 to match your cluster. * Upgrade History: If your cluster was originally installed as an older version (e.g., v3.1) and upgraded to v4.0.9, StarRocks preserves your original defaults to avoid breaking behavior during an upgrade. New defaults in newer versions often only apply to "fresh" installations. * Hardware-Based Auto-Tuning: Some StarRocks parameters (like 
tablet_writer_open_file_limit
or memory-related configs) are calculated dynamically based on the CPU core count and RAM detected when the FE/BE first starts. If your hardware differs from the "standard" lab machine the docs assume, the "default" will look different. 3. How to see the Actual Default for your specific binary Since 
ADMIN SHOW FRONTEND CONFIG
doesn't show the default value column, you can use this trick: 1. Check the Source: In v4.0.9, you can look at the 
IsMutable
 column in 
ADMIN SHOW FRONTEND CONFIG
. If it is 
true
, it is a dynamic config. 2. Search the Logs: When the FE starts up, it logs every configuration value it is using. Search 
fe.log
for the string 
conf field
. It will show the value being loaded. 3. Check the internal 
Config.java
(Advanced): Since StarRocks is open source, for version 4.0, the 
Config.java
file in the GitHub 
branch-4.0
is the ultimate source of truth for that version's defaults. Recommendation for your Memory Issue Since you are investigating memory bloat (the 55M 
SlotRef
objects) and you noticed 
enable_query_queue_v2
is `false`: * The legacy query queue (v1) is what is currently active on your system. * If you have a very high volume of small queries, the legacy queue can sometimes lead to more object overhead in the FE than the newer V2. * However, I would recommend keeping it 
false
 for now while you debug the memory leak, as changing the queuing engine on a live, high-load cluster can significantly change performance characteristics of how queries are admitted. Focus first on the 
max_query_detail_info_num
and 
query_detail_limit
settings discussed earlier. References • administration/management/FE_configuration.md • sql-reference/System_variable.md • administration/management/resource_management/query_queues.md
b
@Rocky could 
enable_mv_query_context_cache
be causing this memory issue?
r
Yes, 
enable_mv_query_context_cache
 can definitely contribute to high FE memory usage, particularly in versions like v4.0.9, if you have complex queries or many Materialized Views (MVs). Why it might be the cause: This parameter (introduced in v3.3) enables a cache for the "Context" of MV rewrites. Its purpose is to speed up the optimizer by reusing results of expensive rewrite logic across different queries. 1. Object Accumulation: The cache stores 
QueryContext
objects. These objects are heavy—they reference the entire query plan, analyzer state, and expression trees, which are made up of millions of 
SlotRef
 and 
Expr
 objects. 2. Memory Multiplier: While the default 
mv_query_context_cache_max_size
is 1000, if your queries are very complex (e.g., hundreds of columns, deeply nested joins), each cached context can easily consume megabytes of RAM. 3. Scope of Cache: In some v4.0 sub-versions, there were reported issues where the "lifetime" of objects within this cache was longer than intended, or the eviction logic didn't account for the deep memory footprint of the expression trees (the 
SlotRefs
). How to verify: You can test this theory immediately without a restart: 1. Disable the cache:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_mv_query_context_cache" = "false");
2. Lower the cache limit (if you want to keep it enabled but safer):
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("mv_query_context_cache_max_size" = "100");
Is this the only cause? While 
enable_mv_query_context_cache
is a candidate, the fact that you have 55 million 
SlotRef
 objects is extremely high for only 13 MVs. Usually, such a massive number points to one of two other things combined with the MV cache: * Query Profile Caching: If 
enable_collect_query_detail_info
is 
true
, the FE stores the detail of every query you run. If you run many queries per second, this is almost always the #1 reason for 
SlotRef
bloat. * MV Refresh "Zombies": If an MV refresh fails or hangs, the "Analyzer" state (which contains thousands of 
SlotRefs
) can sometimes get stuck in memory. Recommended Action Plan: 1. Disable the MV context cache using the command above. 2. Check if 
enable_collect_query_detail_info
is true:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_collect_query_detail_info';
If it is 
true
, set it to 
false
. 3. Monitor Memory: Use 
jmap -histo:live <pid>
or check the 
MemoryUsageTracker
in 
fe.log
. If the 
SlotRef
count starts dropping (after a GC cycle), you've found the culprit. Note: Disabling 
enable_mv_query_context_cache
might slightly increase the "Optimization Time" of your queries that use MVs, but it is a safe trade-off to stabilize the FE memory. References • administration/management/FE_configuration.md • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • faq/fe_mem_faq.md References • administration/management/FE_configuration.md • using_starrocks/caching/query_cache.md • using_starrocks/async_mv/feature-support-asynchronous-materialized-views.md • using_starrocks/async_mv/use_cases/query_rewrite_with_materialized_views.md • sql-reference/System_variable.md
b
@Rocky my memory usage is still increasing even after disabling ``enable_mv_query_context_cache``
r
If your memory is still increasing after disabling the Materialized View (MV) context cache, it indicates that the SlotRef objects (expression trees) are likely being held by another internal system or that the old objects have not been garbage collected. Here are the most likely reasons for 
SlotRef
bloat in StarRocks v4.0.9 and how to address them: 1. Query Detail / Profile Cache (The #1 Culprit) StarRocks stores the execution details (profiles) of recent queries in FE memory. These profiles contain the full expression trees for every query, which can result in millions of 
SlotRef
objects if you have high query concurrency or complex queries. * Check the current limit:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE '%query_detail%';
Look for 
max_query_detail_info_num
(default is often 100). If this is set very high, it will hog memory. * Stop new collection:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("enable_collect_query_detail_info" = "false");
* Force a "Purge": To clear the current memory backlog without a restart, try lowering the limit temporarily to 1, then back to your desired level:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("max_query_detail_info_num" = "1");
    -- Wait 30 seconds for the internal cleanup thread to run
    ADMIN SET FRONTEND CONFIG ("max_query_detail_info_num" = "100");
2. "Zombie" Queries If queries are hung or "stuck" in the FE (even if they appear finished on the client side), their 
ConnectContext
and associated 
SlotRef
objects will stay in the heap forever. * Check for active/hung queries:
Copy code
sql
    SHOW PROC '/current_queries';
    -- and
    SHOW PROCESSLIST;
If you see queries that have been running for hours or days, kill them using 
KILL <connection_id>;
. 3. Statistics Collection Profiles Starting in v3.1+, StarRocks can generate profiles for its internal statistics collection queries. If this is enabled, every background "Analyze" task will add more objects to the profile cache. * Check and disable:
Copy code
sql
    ADMIN SHOW FRONTEND CONFIG LIKE 'enable_statistics_collect_profile';
    -- If true, disable it:
    ADMIN SET FRONTEND CONFIG ("enable_statistics_collect_profile" = "false");
4. Optimizer Expression Cache The optimizer has an internal cache for scalar operators to avoid re-computing complex expressions. If your queries are extremely wide (hundreds of columns) or use massive 
IN
clauses, this cache can grow. * Limit the complexity:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("max_scalar_operator_flat_children" = "5000");
5. Shared-Data Mode Shard Metadata (If using StarOS/Lake) If you are running in Shared-Data (Cloud Native) mode, there is a known issue in some v4.0.x versions where "shards" (metadata for tablets) are not reclaimed properly, leading to millions of 
com.staros.*
objects in the FE heap. * Check metrics: See if 
starrocks_fe_starmgr_num_shards
is in the millions while your actual tablet count is low. If so, this requires a specific patch or a restart to clear. Next Steps for Diagnosis If the memory still doesn't drop after 5–10 minutes: 1. Check the JVM GC: Run 
jcmd <pid> GC.run
to force a full collection and see if the memory stays high. 2. Module Memory Tracker: Run the following to see which internal module StarRocks thinks is using the memory:
Copy code
bash
    grep "Module" fe/log/fe.log | tail -n 20
3. Heap Histogram: If you have access to the shell, run:
Copy code
bash
    jmap -histo:live <pid> | head -n 20
If 
SlotRef
is still at the top, and you have disabled 
enable_collect_query_detail_info
, then a FE Restart is likely necessary to purge the leaked objects, as some internal caches in v4.0.9 do not fully clear until the heap until the process cycles are reset.
Rocky Recommendation: Most users find that disabling 
enable_collect_query_detail_info
and lowering 
max_query_detail_info_num
to 
100
solves the 
SlotRef
bloat immediately. References • knowledge/trouble_shooting/fe_memory_problems.md • knowledge/trouble_shooting/fe_deadlock_problems.md • faq/fe_mem_faq.md
b
@Rocky 
enable_collect_query_detail_info
is already false. 
max_query_detail_info_num
does not appear to be a real value in my version (4.0.9). 
enable_statistics_collect_profile
is also false. there are about 70 items with a large Time value in full processlist all from the same user. i've killed some of them, but others i cannot kill
r
The fact that you have unkillable processes with high "Time" values and a massive count of 
SlotRef
 objects is the smoking gun. In StarRocks, if a process cannot be killed via 
KILL CONNECTION
, it usually means the thread is stuck in a CPU-intensive loop within the FE's Java code (typically the Optimizer) where it isn't checking for the "interrupted" signal. Here is what is happening and how to fix it: 1. Why are the processes unkillable? When you run 
KILL
, StarRocks sets an "isKilled" flag on the connection. However, if the thread is stuck deep inside a complex calculation—like rewriting a massive SQL expression or calculating join distributions for a query with hundreds of joins—it may never reach the point in the code where it checks that flag. Common "Stuck" Points in v4.0.9: * Scalar Operator Rewriting: If a query has a massive 
IN (...)
clause or thousands of 
OR
conditions, the optimizer tries to simplify them. * Materialized View Rewrite: Even if the cache is off, the active attempt to rewrite a query against 13 MVs can consume millions of 
SlotRefs
and get stuck if the query is complex enough. 2. The 
SlotRef
Connection Each 
SlotRef
represents a column reference in an expression. 55 million objects indicate that these 70 "zombie" queries are likely extremely "wide" or "deep" (e.g., 
SELECT * FROM ...
on a table with 1000 columns, multiplied by dozens of join levels). Each query is holding onto its own massive expression tree in the FE heap. 3. Immediate Recovery Steps A. Identify the "Stuck" Code (Critical for RCA) Before you restart, run this from the command line of the FE host to see exactly where those 70 threads are stuck:
Copy code
bash
jstack <FE_PID> > fe_stack.txt
grep -A 20 "starrocks-mysql-nio-pool" fe_stack.txt
Look for threads in the 
RUNNABLE
state. If you see 
com.starrocks.sql.optimizer
or 
ScalarOperator
at the top of the stack, you've confirmed it's an optimizer loop. B. The "Soft" Kill Since 
KILL
didn't work, the only way to reclaim that memory in v4.0.9 is a Restart of the FE. * If you have a High Availability (HA) cluster, the Follower will take over. * Warning: The memory will likely start climbing again as soon as that same user/application resubmits those specific queries. 4. Prevention Settings for v4.0.9 To prevent these queries from ballooning the memory again, set these limits to "fail fast" instead of "bloat memory": 1. Limit Expression Complexity:
Copy code
sql
    -- This limits how many rewrites the optimizer will attempt before giving up
    ADMIN SET FRONTEND CONFIG ("max_planner_scalar_rewrite_num" = "10000");
    -- This limits the width of expression trees
    ADMIN SET FRONTEND CONFIG ("max_scalar_operator_flat_children" = "1000");
2. Lower the Query Timeout: If these are coming from a specific user, lower their session timeout so the FE attempts to stop them sooner (though this may still fail if they are stuck in the optimizer):
Copy code
sql
    SET GLOBAL query_timeout = 300; -- 5 minutes
3. Check for "Stuck" MV Refresh: If that user is an internal system user, check if these "queries" are actually MV refresh tasks:
Copy code
sql
    SHOW MATERIALIZED VIEWS; -- Look for "Active" tasks that have been running for hours
Summary of your Version (4.0.9) In v4.0.9, the parameter is actually named 
query_detail_cache_size
 (instead of 
max_query_detail_info_num
), but since you have 
enable_collect_query_detail_info
set to 
false
, that cache is already inactive. The verdict: Your FE memory is being held by active (but stuck) query threads. You must restart the FE to clear the 55M objects, and you should investigate the SQL being sent by that specific user—it likely contains an "SQL anti-pattern" (like an enormous 
CASE WHEN
or 
IN
list) that is triggering a known optimizer edge case. References
b
@Rocky i see lots of TIMED_WAITING
r
The presence of 
TIMED_WAITING
 threads combined with unkillable processes and millions of 
SlotRef
 objects points to a specific scenario: Your FE threads are stuck waiting for an external resource (likely metadata) and are holding onto massive query analysis states while they wait. Why this is happening: 1. Thread State (
TIMED_WAITING
): In the FE, this usually means a thread is waiting for a response from an external system (like a Hive Metastore, Iceberg Catalog, or Glue) or waiting to acquire a metadata lock. 2. Unkillable Processes: StarRocks' 
KILL
command is "cooperative." It sets a flag that the query thread must check. If a thread is waiting inside a Java library call (like a Hadoop/Hive client call) or stuck in a lock-acquisition loop that doesn't check the interruption flag, it becomes "unkillable" until that operation times out or completes. 3. The 
SlotRef
Connection: 55 million 
SlotRef
objects for 70 queries means each query is holding roughly 780,000 objects. This happens when queries are extremely complex (thousands of columns, deeply nested joins, or massive 
IN
lists). These objects are held by the 
Analyzer
and 
Optimizer
contexts. As long as the thread is "waiting," it cannot release these objects. Immediate Diagnosis Steps: 1. Inspect the "Stuck" Threads Since you see 
TIMED_WAITING
, find out what they are waiting for. Run this on the FE host:
Copy code
bash
# Replace <PID> with your FE process ID
jstack <PID> | grep -A 30 "starrocks-mysql-nio-pool"
Look for keywords like 
com.starrocks.connector
, 
hive
, 
iceberg
, or 
LockManager
. If you see a connector in the stack, your FE is likely waiting for an external catalog that is unresponsive. 2. Check for Lock Contention Since v3.3/v4.0, StarRocks provides a view to see if queries are blocked by metadata locks:
Copy code
sql
SELECT * FROM information_schema.fe_locks WHERE granted = false;
-- OR 
SHOW PROC '/locks';
If you see 70 threads waiting for a lock held by a single thread, you've found the bottleneck. 3. Identify the "Big Queries" The fact that it's the "same user" suggests a specific application or scheduled job is resubmitting these "heavy" queries. * Check the SQL: Run 
SHOW FULL PROCESSLIST
and copy the SQL for one of those 70 queries. Is it an extremely long SQL (many kilobytes/megabytes in size)? * Check External Catalogs: Are these queries hitting a Hive/Iceberg/JDBC catalog? If the catalog is slow, every retry by the user adds another ~1GB of 
SlotRef
objects to the heap. Recommended Fixes: 1. Reclaim Memory (Restart Required): Because these threads are unkillable and holding onto 55M objects, your FE will likely OOM soon. You must restart the FE to clear the "zombie" contexts. 2. Enable the "Analyzer" Timeout: To prevent queries from holding memory indefinitely during analysis:
Copy code
sql
    -- This helps limit the time spent in the optimizer
    ADMIN SET FRONTEND CONFIG ("optimizer_timeout_stat_sample" = "5000");
3. Check/Disable Automatic Statistics: Sometimes a "Statistics" job can get stuck if a table is massive.
Copy code
sql
    -- Check if there are running stats tasks
    SHOW ANALYZE;
    -- Consider disabling it if it's the cause
    ADMIN SET FRONTEND CONFIG ("enable_statistic_collect" = "false");
4. Identify the User/App: Since all 70 items are from the same user, you need to stop that application from retrying. If it sees a timeout, it might be resubmitting immediately, creating a "Query Storm" that fills your heap with 
SlotRefs
. Rocky's Tip: Check the folder 
fe/log/proc_profile/
. In v4.0.9, StarRocks automatically generates memory flame graphs (
.html.tgz
files). Download the latest one—it will show you exactly which module (Analyzer, Optimizer, or Catalog) is owning those 55 million 
SlotRef
objects. References
b
@Rocky this is the jmap after an FE restart root@prod-starrocks-fe-2:/opt/starrocks# jmap -histo:live 19 | head -n 20 num #instances #bytes class name (module) ------------------------------------------------------- 1: 8628270 1092480232 [B (java.base@17.0.18) 2: 4085854 784483968 com.starrocks.analysis.SlotRef 3: 5090612 285074272 com.starrocks.sql.analyzer.Field 4: 8806304 281801728 java.lang.String (java.base@17.0.18) 5: 8707887 278652384 com.google.common.collect.ImmutableMapEntry 6: 2706240 273335616 [Ljava.lang.Object; (java.base@17.0.18) 7: 8737153 209691672 com.google.common.collect.SingletonImmutableList 8: 5091500 203660000 java.util.LinkedList (java.base@17.0.18) 9: 1372064 193938992 [Ljava.util.HashMap$Node; (java.base@17.0.18) 10: 5820661 186261152 java.util.ArrayList (java.base@17.0.18) 11: 2928803 164012968 java.util.LinkedHashMap$Entry (java.base@17.0.18) 12: 3898993 155959720 com.google.common.collect.ImmutableMapEntry$NonTerminalImmutableMapEntry 13: 69740 155091520 [Lcom.google.common.collect.ImmutableMapEntry; 14: 1531574 122525920 java.util.LinkedHashMap (java.base@17.0.18) 15: 4929480 118307520 java.lang.Long (java.base@17.0.18) 16: 1309925 104794000 com.starrocks.lake.LakeTablet 17: 2482064 99282560 java.util.HashMap$Node (java.base@17.0.18) 18: 40138 69519200 [Ljava.util.Map$Entry; (java.base@17.0.18)
r
Sorry, I encountered an error while processing your request: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Resource has been exhausted (e.g. check quota).', 'status': 'RESOURCE_EXHAUSTED'}}
Open in Slack
PreviousNext
Powered by