http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Can you assist in some troubleshoti...
# questions-and-troubleshooting
a
@Rocky Can you assist in some troubleshoting about memory exhaustion on the FE node, compounded by a saturated cache-dict thread pool?
r
Troubleshooting memory exhaustion on the Frontend (FE) node, especially when paired with a saturated 
cache-dict
thread pool, usually points to issues with the Global Dictionary (Low Cardinality Optimization) mechanism. 1. Understanding the 
cache-dict
Thread Pool The 
cache-dict
thread pool in StarRocks is responsible for loading and refreshing Global Dictionaries from the Backend (BE) nodes into the FE's memory. * What it does: When the optimizer decides to use low-cardinality optimization (encoding string columns as integers), the FE must fetch the dictionary to perform plan rewriting. * Saturation: If you see "rejected execution" logs for the 
cache-dict
pool, it means the FE is trying to load too many dictionaries simultaneously, exceeding the pool's capacity (which is often fixed at 16 threads in some versions). * Memory Impact: These dictionaries are cached in the FE heap. If your workload involves many tables, columns, or partitions with low-cardinality strings, the cumulative size of these dictionaries can exhaust the FE's JVM heap. 2. Immediate Troubleshooting Steps Step 1: Identify the Memory Consumer Use the internal memory tracker to see which module is consuming the most heap.
Copy code
sql
-- Available in StarRocks v3.3 and later
SHOW PROC '/memory_usage';
Look for modules named "Dictionary", "GlobalDict", or "Metadata". If "Dictionary" is high, the global dictionary cache is the primary culprit. Step 2: Check for Thread Pool Rejections Search your 
fe.log
or 
fe.warn.log
for the following pattern: 
Task ... rejected from cache-dict ... pool size = 16
If this is frequent, your query concurrency or the sheer number of dictionaries being loaded is overwhelming the FE. Step 3: Mitigate via Configuration If the FE is at risk of crashing (OOM), you can temporarily disable the low-cardinality optimization. This will stop the FE from loading new dictionaries into memory.
Copy code
sql
-- Disable globally (requires restart or applies to new sessions)
SET GLOBAL cbo_enable_low_cardinality_optimize = false;
Note: This might slightly increase query latency for string-heavy queries, but it will stabilize the FE memory. 3. Root Cause Analysis & Solutions Potential Cause | Solution Too many optimized columns | Check your tables for columns with cardinality just under the threshold (default 255). If you have hundreds of such columns across many partitions, the FE memory will bloat. High Partition Count | Queries scanning thousands of partitions, each requiring a dictionary load, can saturate the thread pool. Consider reducing partition granularity or using 
TRUNCATE
instead of many small deletes/loads. Insufficient FE Heap | If your metadata and dictionaries simply outgrow the default heap, increase the 
-Xmx
value in 
fe.conf
. For large clusters, 16GB–32GB is common. Stale Dictionaries | In some versions, dictionaries might not be evicted fast enough. Upgrading to the latest maintenance release (v3.2.x or v3.3.x) often includes fixes for dictionary cache lifecycle management. 4. Advanced Tuning If you must keep the optimization enabled but need to reduce the "frenzy" of the 
cache-dict
pool: * Lower the Threshold: You can reduce the 
low_cardinality_threshold
(default 255) so fewer columns qualify for the optimization.
Copy code
sql
    -- In fe.conf (requires restart)
    low_cardinality_threshold = 128
* Analyze Statistics: Ensure your statistics are up to date. The optimizer uses cardinality stats to decide whether to trigger a dictionary load. If stats are missing or stale, it might make suboptimal caching decisions. Recommended Next Step: Run 
jmap -histo <FE_PID> | head -n 20
to see if 
com.starrocks.sql.optimizer.statistics.CacheDict
or similar objects are dominating the heap. If you see high counts of these objects, the Global Dictionary cache is definitely the bottlenecked. References
Your analysis is extremely precise and correctly identifies the Global Dictionary (Low Cardinality Optimization) as the primary driver of your Frontend (FE) instability. The sequence you described is a classic "GC Death Spiral": the heap is dominated by long-lived dictionary objects, leaving no room for short-lived task metadata (like 
PUBLISH_VERSION
contexts), which causes thread pool saturation and eventual system unresponsiveness. 1. Root Cause: Global Dictionary Bloat The 
CacheDictManager
is responsible for caching string-to-integer mappings for low-cardinality columns to speed up queries. Having 30+ optimized columns per table is very high. With high transaction throughput, the FE is likely loading and maintaining a massive number of distinct dictionary versions across many partitions. 2. Immediate Mitigation (Stabilization) To prevent the FE from crashing or becoming completely unresponsive, you should disable the Low Cardinality Optimization globally. This will stop the FE from attempting to load new dictionaries into the heap, allowing the G1GC to reclaim space as old dictionaries expire.
Copy code
sql
-- Apply globally to stop the FE from loading new dictionaries
SET GLOBAL cbo_enable_low_cardinality_optimize = false;
Note: This will not break queries; it simply reverts the optimizer to using standard string processing instead of the dictionary-encoded integer path. 3. Tuning the Dictionary Thresholds If you want to re-enable the feature later, you must reduce the number of columns that qualify for it. * Version 3.5.0+: Adjust the threshold in `fe.conf`:
Copy code
properties
    # Reduce from default 255 to something smaller like 64 or 128
    low_cardinality_threshold = 64
* Earlier Versions: You may need to manually identify and disable the optimization for specific columns by altering the table or using session variables to control the rewrite. 4. Handling the 
cache-dict
Thread Pool Saturation The saturation (184k rejections) is largely a side-effect of the JVM being "stunned" by high GC overhead. However, if your workload genuinely requires high-concurrency dictionary loading, check if your version supports the internal config 
cache_dict_thread_pool_size
(this is often internal/hidden). * Run 
ADMIN SHOW FRONTEND CONFIG LIKE "%dict%";
to see if your version allows manual expansion of this pool. 5. Memory Recovery & Monitoring * Increase Heap: For a high-throughput cluster with heavy metadata and dictionary usage, 32GB is often the "danger zone." If your hardware permits, consider increasing 
-Xmx
to 64GB. * Detailed Memory Breakdown: If you are on v3.3+, use the internal tracker to confirm the dictionary size:
Copy code
sql
    SHOW PROC '/memory_usage';
Look for the "Dictionary" or "CacheDictManager" modules. If they exceed 10-15GB, you have a clear over-caching problem. * Manual Cleanup: Currently, there is no SQL command to "flush" the internal Global Dictionary cache specifically. Disabling the feature (
cbo_enable_low_cardinality_optimize = false
) and waiting for LRU eviction or a rolling FE restart is the standard procedure. 6. Cascading Issues (
PUBLISH_VERSION
and Thrift) * Orphaned Tasks: The 
cannot find task. type: PUBLISH_VERSION
errors occur because the FE's internal task map is likely being cleared or the FE is so slow that the BE's report-back exceeds the FE's internal timeout/retention. * Thrift Threads: The 16,000+ thread count is extremely high (the default max is usually 4,096). This indicates that RPC requests are "piling up" because the FE is stuck in GC pauses and cannot complete the work, forcing the Thrift server to spawn more threads to handle new incoming connections until it hits OS limits. Recommendation: 1. Disable 
cbo_enable_low_cardinality_optimize
. 2. Perform a rolling restart of FEs to clear the "ghost" Thrift threads and reset the heap.
3. Increase FE Heap to 48G or 64G if the dictionary usage remains a requirement. high. References • administration/management/FE_configuration.md
Open in Slack
PreviousNext
Powered by