http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> One cn pod is running on 90%+ memo...
# questions-and-troubleshooting
h
@Rocky One cn pod is running on 90%+ memory usages can you help me debug have fetched dump snapshot [which was enabled 12-15 hrs back]
Copy code
StarRocks
MemTracker
Memory
Configs
Memory Usage Detail
level	Label	Parent	Limit	Current Consumption	Peak Consumption
1	process		44G	43G	45G
2	query_pool	process	40G	5G	23G
2	load	process	13G	0	5G
2	compaction	process	44G	0	8G
2	schema_change	process	none	0	0
2	column_pool	process	none	0	0
2	page_cache	process	none	260M	8G
2	jit_cache	process	none	6K	6K
2	chunk_allocator	process	none	292M	2G
2	clone	process	none	0	0
2	consistency	process	8G	0	0
2	datacache	process	none	1484M	1577M
2	replication	process	none	0	0
2	metadata	process	none	643M	965M
2	update	process	26G	2G	4G
 version 3.3.2-857dd73
BuildType: RELEASE
Untitled
@Rocky ? ^
r
Hello! I'm Rocky, your StarRocks assistant. It looks like your message might have been cut off. How can I help you with StarRocks today? You can ask me about: * SQL syntax and functions * Table design and data loading * Performance tuning and optimization * Cluster configuration and administration
From the snippet you provided for StarRocks v3.3.2, the most striking detail is the gap between the 
process
memory (43G) and the sum of the currently tracked modules (which adds up to only ~10G). Here is how to debug this "missing" memory on your CN node: 1. Check Allocator Cache (The most likely cause) Your 
query_pool
peak was 23G, and 
page_cache
peak was 8G, but they have dropped to 5G and 260M respectively. Often, the memory allocator (TCMalloc or Jemalloc) does not immediately return this memory to the Operating System, keeping it in "freelists" for future use. * Action: Access the memory allocator status via the browser: 
http://<cn_ip>:<cn_http_port>/memz
* What to look for: * If you see a large value in 
Bytes in page heap freelist
 (for TCMalloc) or 
stats.metadata
/ 
stats.resident
 (for Jemalloc), then the memory is held by the allocator's cache, not a specific StarRocks module. * StarRocks v3.3 uses Jemalloc by default. Check if 
enable_jemalloc_memory_tracker
is 
true
in your CN config (it should be in 3.3.2). 2. Investigate Data Cache Metadata On CN nodes, the Data Cache (Block Cache) can consume significant memory for metadata that isn't always perfectly captured in the top-level 
datacache
tracker. * Action: Check the detail of the data cache: 
curl http://<cn_ip>:<cn_http_port>/metrics | grep starlet_cache
* If the metadata size is huge, it might be due to a high number of small files/cached blocks. 3. Drill Down the MemTracker The summary you provided is at 
level 2
. There might be untracked allocations or specific sub-trackers holding memory. * Action: Get a more granular view: 
http://<cn_ip>:<cn_http_port>/mem_tracker?type=process&upper_level=4
* Look for any label that doesn't belong to the categories you listed. 4. Jemalloc Heap Profiling (Since you have a snapshot) If you "fetched a dump snapshot" and it's a Jemalloc Heap Profile (usually a 
.heap
file), you can analyze it to see exactly which code path allocated the memory that hasn't been freed. * Action: Use 
jeprof
to visualize it:
Copy code
bash
    jeprof --pdf ./starrocks_be ./jeprof.12345.0.f.heap > profile.pdf
_(Note: Replace 
starrocks_be
with your actual CN binary and the 
.heap
file with your snapshot)._ Summary of the Situation Based on your 
Peak Consumption
vs `Current Consumption`: * The system previously reached 45G (over the 44G limit), likely triggered by a combination of a large query (23G) and page cache (8G). * Even though those tasks finished, the process is still holding 43G. * Recommendation: If the memory isn't being released and it's causing OOM risks for new queries, you can try to trigger a manual release (though StarRocks usually manages this) or restart the CN pod if it's a persistent leak. In 3.3.2, check if there are many 
starlet
related errors in the 
<http://be.INFO|be.INFO>
log, as metadata leaks in the cache were fixed in later 3.3.x patches. Would you like me to help analyze a specific section of the 
<http://be.INFO|be.INFO>
logs or the 
/memz
output? References • knowledge/trouble_shooting/fe_memory_problems.md • using_starrocks/caching/block_cache.md • faq/fe_mem_faq.md • administration/management/resource_management/Memory_management.md • knowledge/usage/best-practice.md
h
@Rocky see this also
Copy code
[hdfcbank@ip-10-196-238-120 ~]$ kubectl -n starrocks exec reporting-1-cn-2 -- curl -s <http://localhost:8040/mem_tracker> <!DOCTYPE html> <html> <head> <title>StarRocks</title> <meta charset='utf-8'/> <link href='/bootstrap/css/bootstrap.min.css' rel='stylesheet' media='screen' /> <link href='/bootstrap/css/bootstrap-table.min.css' rel='stylesheet' media='screen' /> <script src='/jquery-3.5.0.min.js' defer></script> <script src='/bootstrap/js/bootstrap.min.js' defer></script> <script src='/bootstrap/js/bootstrap-table.min.js' defer></script> <script src='/starrocks.js' defer></script> <link href='/starrocks.css' rel='stylesheet' /> </head> <body> <nav class="navbar navbar-default"> <div class="container-fluid"> <div class="navbar-header"> <a class="navbar-brand" style="padding-top: 5px;" href="/"> <img src="/starrocks-logo.png" width='40' height='40' alt="StarRocks" /> </a> </div> <div id="navbar" class="navbar-collapse collapse"> <ul class="nav navbar-nav"> <li><a class="nav-link"href="/mem_tracker">MemTracker</a></li> <li><a class="nav-link"href="/memz">Memory</a></li> <li><a class="nav-link"href="/varz">Configs</a></li> </ul> </div><!--/.nav-collapse --> </div><!--/.container-fluid --> </nav> <h1>Memory Usage Detail</h1> <table data-toggle='table' data-pagination='true' data-search='true' class='table table-striped'> <thead><tr><th>level</th><th>Label</th><th>Parent</th><th>Limit</th><th data-sorter='bytesSorter' data-sortable='true' >Current Consumption</th><th data-sorter='bytesSorter' data-sortable='true' >Peak Consumption</th><tbody> <tr><td>1</td><td>process</td><td></td><td>44G</td><td>43G</td><td>45G</td></tr> <tr><td>2</td><td>query_pool</td><td>process</td><td>40G</td><td>5G</td><td>23G</td></tr> <tr><td>2</td><td>load</td><td>process</td><td>13G</td><td>0</td><td>5G</td></tr> <tr><td>2</td><td>compaction</td><td>process</td><td>44G</td><td>0</td><td>8G</td></tr> <tr><td>2</td><td>schema_change</td><td>process</td><td>none</td><td>0</td><td>0</td></tr> <tr><td>2</td><td>column_pool</td><td>process</td><td>none</td><td>0</td><td>0</td></tr> <tr><td>2</td><td>page_cache</td><td>process</td><td>none</td><td>260M</td><td>8G</td></tr> <tr><td>2</td><td>jit_cache</td><td>process</td><td>none</td><td>6K</td><td>6K</td></tr> <tr><td>2</td><td>chunk_allocator</td><td>process</td><td>none</td><td>292M</td><td>2G</td></tr> <tr><td>2</td><td>clone</td><td>process</td><td>none</td><td>0</td><td>0</td></tr> <tr><td>2</td><td>consistency</td><td>process</td><td>8G</td><td>0</td><td>0</td></tr> <tr><td>2</td><td>datacache</td><td>process</td><td>none</td><td>1484M</td><td>1577M</td></tr> <tr><td>2</td><td>replication</td><td>process</td><td>none</td><td>0</td><td>0</td></tr> <tr><td>2</td><td>metadata</td><td>process</td><td>none</td><td>643M</td><td>965M</td></tr> <tr><td>2</td><td>update</td><td>process</td><td>26G</td><td>2G</td><td>4G</td></tr> </tbody></table> </div> <footer class="footer"><div class="container text-muted"> <pre> version 3.3.2-857dd73 BuildType: RELEASE</pre> </div></footer> </body> </html> [hdfcbank@ip-10-196-238-120 ~]$ [hdfcbank@ip-10-196-238-120 ~]$ kubectl -n starrocks exec reporting-1-cn-2 -- curl -s <http://localhost:8040/metrics> | grep -E "^starrocks_be_.*_mem_bytes" starrocks_be_bitmap_index_mem_bytes 0 starrocks_be_bloom_filter_index_mem_bytes 1174704 starrocks_be_chunk_allocator_mem_bytes 2054426040 starrocks_be_clone_mem_bytes 0 starrocks_be_column_metadata_mem_bytes 611340060 starrocks_be_column_pool_mem_bytes 0 starrocks_be_column_zonemap_index_mem_bytes 100413964 starrocks_be_compaction_mem_bytes 0 starrocks_be_consistency_mem_bytes 0 starrocks_be_datacache_mem_bytes 1556595096 starrocks_be_jit_cache_mem_bytes 6952 starrocks_be_load_mem_bytes 408191168 starrocks_be_metadata_mem_bytes 674517324 starrocks_be_ordinal_index_mem_bytes 285745504 starrocks_be_process_mem_bytes 44202303528 starrocks_be_query_mem_bytes 1252246184 starrocks_be_rowset_metadata_mem_bytes 0 starrocks_be_schema_change_mem_bytes 0 starrocks_be_segment_metadata_mem_bytes 63136094 starrocks_be_segment_zonemap_mem_bytes 46741600 starrocks_be_short_key_index_mem_bytes 3904542 starrocks_be_storage_page_cache_mem_bytes 273678512 starrocks_be_tablet_metadata_mem_bytes 41170 starrocks_be_tablet_schema_mem_bytes 41170 starrocks_be_update_mem_bytes 2807886775 [hdfcbank@ip-10-196-238-120 ~]$
sample table
Copy code
| ENTITY_transactions_cms | CREATE TABLE `ENTITY_transactions_cms` (
  `id` varchar(65533) NOT NULL COMMENT "",
  `timestamp` datetime NOT NULL COMMENT "",
  `count` tinyint(4) NULL DEFAULT "1" COMMENT "",
  `_pr_transactionId` varchar(65533) NULL COMMENT "",
  `_pr_transactionType` varchar(65533) NULL COMMENT "",
  `_pr_purpose__purposeVersion2` bigint(20) NULL COMMENT "",
  `_pr_purpose__purposePreference__preferenceId` varchar(65533) NULL COMMENT "",
  `_pr_purpose__purposePreference__options` json NULL COMMENT "",
  `_pr_purpose__purposeId` varchar(65533) NULL COMMENT "",
  `_pr_identifier` varchar(65533) NULL COMMENT "",
  `_pr_isMigrated` boolean NULL COMMENT "",
  `_pr_receiptId` varchar(65533) NULL COMMENT "",
  `_pr_expirtyDate` bigint(20) NULL COMMENT "",
  `createdTime` bigint(20) NULL COMMENT "",
  `modifiedTime` bigint(20) NULL COMMENT "",
  `version` bigint(20) NULL COMMENT "",
  `deleted` boolean NULL COMMENT "",
  `ownerUserId` bigint(20) NULL COMMENT "",
  `lastModifiedBy` bigint(20) NULL COMMENT "",
  `projectId` bigint(20) NULL COMMENT "",
  `grants` varchar(65533) NULL COMMENT "",
  `_pr_purpose__purposeName` varchar(65533) NULL COMMENT "",
  `_pr_fileName` varchar(65533) NULL COMMENT "",
  `_pr_migratedTime` bigint(20) NULL COMMENT "",
  `translations` json NULL COMMENT ""
) ENGINE=OLAP
PRIMARY KEY(`id`, `timestamp`)
COMMENT "OLAP"
PARTITION BY time_slice(timestamp, 7, 'day', 'floor')
DISTRIBUTED BY HASH(`id`) BUCKETS 32
PROPERTIES (
"bloom_filter_columns" = "_pr_identifier",
"compression" = "LZ4",
"datacache.enable" = "true",
"datacache.partition_duration" = "1 months",
"enable_async_write_back" = "false",
"enable_persistent_index" = "true",
"persistent_index_type" = "LOCAL",
"replication_num" = "1",
"storage_volume" = "s3_analytics_volume"
); |
r
Based on the 
mem_tracker
and 
metrics
data you provided for StarRocks v3.3.2, here is the breakdown of your memory usage and how to debug the discrepancy. 1. The "Missing" Memory Gap (~33GB) Your 
process
memory is 43G, but the sum of your tracked modules (
query_pool
+ 
load
+ 
datacache
+ 
update
+ 
metadata
) is only about 10G. There is a 33GB discrepancy. Diagnosis: This is almost certainly retained memory held by the memory allocator (Jemalloc). * The Evidence: Your 
Peak Consumption
columns show that 
query_pool
previously hit 23G, 
page_cache
hit 8G, and 
load
hit 5G. The sum of these peaks is roughly 40-45G. * What happened: At some point in the last 12-15 hours, a heavy workload (large queries or loads) caused the process to allocate 45G. When those tasks finished, StarRocks "freed" the memory, but the Jemalloc allocator kept it in its internal caches (freelists/retained memory) to reuse for future requests instead of giving it back to the OS. * K8s Impact: Kubernetes and 
kubectl top
see the memory held by the allocator as "used" by the pod, even if StarRocks' internal trackers say it's free. 2. Immediate Debugging Actions To confirm the allocator's state, check the 
/memz
endpoint (which for Jemalloc provides low-level stats):
Copy code
bash
# Exec into the pod and curl the memz endpoint
kubectl -n starrocks exec reporting-1-cn-2 -- curl -s <http://localhost:8040/memz>
Look for these specific fields in the output: * 
retained
: Memory held by jemalloc that is unmapped from the application but still part of the virtual address space. * 
dirty
: Memory that has been freed by the application but not yet purged/returned to the OS. 3. Primary Key Table & CN Persistence Your DDL shows a Primary Key table with 
"persistent_index_type" = "LOCAL"
. * On CN Nodes: Since CN nodes are typically ephemeral in K8s, ensure you have a persistent volume (SSD) mounted to the storage path. If the local disk is slow or full, the index management can become unstable. * Memory Usage: Although your 
update
tracker shows only 2G, remember that PK tables on CNs in v3.3.2 had several known metadata tracking issues. If you see high 
metadata
or untracked growth after long periods of loading, it may be related to the 
starlet
cache metadata. 4. Recommendations & Fixes A. Force Memory Release You can try to tell Jemalloc to return memory to the OS by decreasing the decay time (this will trigger more aggressive purging):
Copy code
bash
# This is a temporary test, not a permanent config change
curl -X POST "http://<cn_ip>:8040/varz/update_config?je_dirty_decay_ms=5000"
curl -X POST "http://<cn_ip>:8040/varz/update_config?je_muzzy_decay_ms=5000"
B. Analyze your Heap Profile Since you already have a dump snapshot (Jemalloc 
.heap
file), use 
jeprof
to see if the "missing" 33G is actually leaked by a specific function or just sitting in the allocator:
Copy code
bash
# If you have the binary and the heap file
jeprof --pdf ./starrocks_be ./your_dump.heap > profile.pdf
If the resulting PDF shows a large "root" node with no clear children, the memory is definitely just sitting in the allocator's cache. C. Version Warning (v3.3.2) You are on a very early version of the 3.3 branch. There is a known Bug #67607 (and several PK memory fixes) in StarRocks 3.3.x where memory statistics in container environments can be inaccurate and Jemalloc retains memory too aggressively. * Action: It is highly recommended to upgrade to at least v3.3.8 or the latest v3.3.13, which includes significant optimizations for Primary Key table memory management and more accurate memory tracking in K8s.
Summary: Your CN pod hit a peak load earlier, and the memory is now "trapped" in the allocator's cache. Unless you see the 
Current Consumption
of 
query_pool
or 
update
climbing continuously without a previous peak, this is likely not a leak but a side effect of Jemalloc's default behavior in-container behavior in v3.3.2. References • administration/management/resource_management/Memory_management.md • faq/fe_mem_faq.md • knowledge/trouble_shooting/fe_memory_problems.md
h
@Rocky
Untitled
r
Hello, @Harit Rajkotiya! How can I assist you today?
h
@Rocky
Copy code
StarRocks
MemTracker
Memory
Configs
Mem Limit: 44.55 GB
Mem Consumption: 40.20 GB
___ Begin jemalloc statistics ___
Version: "5.3.0-0-g54eaed1d8b56b1aa528be3bdd1877e59c56fa90c"
Build-time option settings
  config.cache_oblivious: true
  config.debug: false
  config.fill: true
  config.lazy_lock: false
  config.malloc_conf: ""
  config.opt_safety_checks: false
  config.prof: true
  config.prof_libgcc: true
  config.prof_libunwind: false
  config.stats: true
  config.utrace: false
  config.xmalloc: false
Run-time option settings
  opt.abort: false
  opt.abort_conf: false
  opt.cache_oblivious: true
  opt.confirm_conf: false
  opt.retain: true
  opt.dss: "secondary"
  opt.narenas: 16
  opt.percpu_arena: "percpu"
  opt.oversize_threshold: 0
  opt.hpa: false
  opt.hpa_slab_max_alloc: 65536
  opt.hpa_hugification_threshold: 1992294
  opt.hpa_hugify_delay_ms: 10000
  opt.hpa_min_purge_interval_ms: 5000
  opt.hpa_dirty_mult: "0.25"
  opt.hpa_sec_nshards: 4
  opt.hpa_sec_max_alloc: 32768
  opt.hpa_sec_max_bytes: 262144
  opt.hpa_sec_bytes_after_flush: 131072
  opt.hpa_sec_batch_fill_extra: 0
  opt.metadata_thp: "auto"
  opt.mutex_max_spin: 600
  opt.background_thread: true (background_thread: true)
  opt.dirty_decay_ms: 5000 (arenas.dirty_decay_ms: 5000)
  opt.muzzy_decay_ms: 5000 (arenas.muzzy_decay_ms: 5000)
  opt.lg_extent_max_active_fit: 6
  opt.junk: "false"
  opt.zero: false
  opt.tcache: true
  opt.tcache_max: 32768
  opt.tcache_nslots_small_min: 20
  opt.tcache_nslots_small_max: 200
  opt.tcache_nslots_large: 20
  opt.lg_tcache_nslots_mul: 1
  opt.tcache_gc_incr_bytes: 65536
  opt.tcache_gc_delay_bytes: 0
  opt.lg_tcache_flush_small_div: 1
  opt.lg_tcache_flush_large_div: 1
  opt.thp: "default"
  opt.prof: true
  opt.prof_prefix: "jeprof"
  opt.prof_active: false (prof.active: true)
  opt.prof_thread_active_init: true (prof.thread_active_init: true)
  opt.lg_prof_sample: 19 (prof.lg_sample: 19)
  opt.prof_accum: false
  opt.lg_prof_interval: -1
  opt.prof_gdump: false
  opt.prof_final: false
  opt.prof_leak: false
  opt.prof_leak_error: false
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_print: false
  opt.stats_print_opts: ""
  opt.stats_interval: -1
  opt.stats_interval_opts: ""
  opt.zero_realloc: "free"
Profiling settings
  prof.thread_active_init: true
  prof.active: true
  prof.gdump: false
  prof.interval: 0
  prof.lg_sample: 19
Arenas: 16
Quantum size: 16
Page size: 4096
Maximum thread-cached size class: 32768
Number of bin size classes: 36
Number of thread-cache bin size classes: 41
Number of large size classes: 196
Allocated: 43409155880, active: 44284780544, metadata: 1808682704 (n_thp 857), resident: 46153674752, mapped: 46451863552, retained: 286676013056
Count of realloc(non-null-ptr, 0) calls: 0
r
The 
jemalloc
statistics provide the "smoking gun" for your memory usage. Here is the analysis: 1. The Discrepancy: Tracked vs. Allocated The most important takeaway from your 
memz
output is: * Jemalloc Allocated: 43.4 GB (This is memory the application has actively requested and is currently holding). * StarRocks MemTracker: ~10 GB (The sum of 
query_pool
, 
load
, 
update
, etc., from your previous snippet). Conclusion: You have approximately 33 GB of "Untracked Memory." This is memory that StarRocks has allocated via 
malloc()
or 
new
, but the code path doing the allocation is not wrapping it in a StarRocks 
MemTracker
. 2. Analysis of the 
memz
Stats * Resident (46.15 GB): This is the actual physical memory (RSS) being used. It matches the "90%+" usage you see in Kubernetes. * Allocated (43.4 GB): Since this is very close to the Resident memory, it means the memory is not just sitting idle in the allocator's cache—it is actively occupied by data structures in the StarRocks process. * Retained (267 GB): This is Virtual Address space. It is normal for Jemalloc with 
opt.retain: true
to have a high VIRT value; this does not impact physical RAM. * Metadata (1.8 GB): Jemalloc's own internal metadata is reasonably low. 3. Likely Culprits in v3.3.2 In StarRocks v3.3.2, there are two primary areas where untracked memory often accumulates, especially on CN nodes using Primary Key tables: A. Block Cache (Starlet) Metadata You have 
datacache.enable = true
and a PK table. The 
datacache
tracker only tracks the data blocks themselves. However, the metadata (index of what is in the cache) was not fully tracked in early 3.3.x versions. If you have many small files or a very large cache, this can grow to tens of gigabytes. * Check this: Run 
curl -s <http://localhost:8040/metrics> | grep starlet_cache_metadata_bytes
. B. Persistent Index (PK Table) Your table uses 
enable_persistent_index = true
. While the 
update
tracker shows some usage, v3.3.2 had issues where certain memory-mapped files or internal buffers for the Persistent Index were not accounted for in the 
MemTracker
. 4. How to identify the exact code path Since you mentioned you have a dump snapshot (specifically a Jemalloc profile), this is the only way to see the 33GB of untracked memory. Run this command on your machine (not inside the pod, as you need the 
jeprof
tool and the 
starrocks_be
binary):
Copy code
bash
# You need the starrocks_be binary from the container and the .heap file
jeprof --pdf ./starrocks_be ./jeprof.xxxx.heap > memory_analysis.pdf
Look for the largest boxes in the PDF. Since the memory is "untracked," the labels in the PDF will show C++ function names (like 
starlet::...
or 
StorageEngine::...
) instead of StarRocks MemTracker labels. Recommendation 1. Analyze the Heap Profile: This will tell you exactly which function allocated the 33GB. 2. Upgrade: StarRocks v3.3.2 is quite old. There have been critical fixes for memory tracking and leaks in the Primary Key/Cloud-Native engine in later versions. Upgrading to v3.3.13 (the latest in that branch) is highly recommended and would likely resolve this "missing" memory issue. If you can share the top few lines of the 
jeprof
text output (
jeprof --text ...
), I can tell you exactly which module is responsible.
4 Views
Open in Slack
PreviousNext
Powered by