@Rocky I have observed untracked memory in cn causing memory exceeded on every query

Memory Usage Detail
level	Label	Parent	Limit	Current Consumption	Peak Consumption
1	process		44G	45G	45G
2	jemalloc_metadata	process	none	1574M	1599M
2	query_pool	process	40G	-288M	15G
2	load	process	13G	0	258M
2	compaction	process	44G	0	5G
2	schema_change	process	none	0	6M
2	column_pool	process	none	0	0
2	page_cache	process	none	261M	8G
2	jit_cache	process	none	21K	21K
2	chunk_allocator	process	none	0	128M
2	clone	process	none	0	0
2	consistency	process	8G	0	0
2	datacache	process	none	1086M	1240M
2	replication	process	none	0	0
2	metadata	process	none	782M	1139M
2	update	process	26G	664M	2G
 version 3.3.9-dfae8f9
BuildType: RELEASE

what to debug further ?

The "untracked memory" you are seeing (where the process usage of 45G far exceeds the sum of level 2 trackers, which total only ~4G in your output) indicates that memory is being allocated by the process but not being captured by StarRocks' internal memory tracking hooks. The negative value for

query_pool

(

-288M

) is a strong indicator of a memory tracking bug (where memory is released more than it was recorded as consumed), which often results in the "missing" memory remaining under the root process tracker. To debug this further on version 3.3.9, follow these steps: 1. Check Jemalloc/Tcmalloc Status (

/memz

) StarRocks uses a memory allocator (usually jemalloc) that may retain memory even after it is "freed" by the application. * Access the endpoint:

http://<CN_IP>:<CN_HTTP_PORT>/memz

* Look for:

Bytes in page heap freelist

(or similar for jemalloc). If this value is very high (e.g., 30-40GB), it means the allocator is holding onto memory instead of returning it to the OS. * Action: Try triggering a manual release by running:

curl -XGET http://<CN_IP>:<CN_HTTP_PORT>/be_cloud_native_test/clear_cache

(Note: command varies, but jemalloc usually has an internal purge mechanism). 2. Enable Jemalloc Heap Profiling (Recommended) Since the memory is untracked by StarRocks trackers, you need a system-level view of what is allocating memory. * Enable profiling:

sql
    ADMIN EXECUTE ON <cn_id> 'System.print(HeapProf.getInstance().enable_prof())';

* Wait for the memory to grow or run a query that triggers the OOM. * Dump the profile:

sql
    ADMIN EXECUTE ON <cn_id> 'System.print(HeapProf.getInstance().dump_dot_snapshot())';

* Analyze: Copy the output (DOT format) and paste it into GraphvizOnline to see the flame graph of which C++ functions are actually holding the 40GB+. 3. Verify Data Cache Metadata In CN nodes, the Data Cache (block cache) can sometimes consume significant memory for metadata that might not be fully accounted for in the level 2

datacache

tracker. * Check metrics:

curl -s http://<CN_IP>:<CN_HTTP_PORT>/metrics | grep -E "star_cache_meta_memory|star_cache_data_memory"

* If meta memory is high, this was a known issue in earlier 3.x versions. Although 3.3.9 contains many fixes, a leak in the cache metadata can still cause "untracked" bloat. 4. Investigate the Negative

query_pool

The negative

query_pool

value is likely caused by a tracking mismatch. In version 3.3.x, there were known issues (e.g., #70228) where memory tracking for certain operators or during data ingestion could become inconsistent. * Check logs: Search

cn/log/cn.WARNING

for any "Memory tracker" related warnings. * Test Isolation: Does this happen only after a specific type of query (e.g., wide joins, complex window functions, or external table queries)? Tracking bugs are often specific to certain execution paths. Summary of Debugging Commands Goal | Command / URL Allocator details |

http://<CN_IP>:8040/memz

Fine-grained trackers |

http://<CN_IP>:8040/mem_tracker?type=query_pool&upper_level=3

Data Cache metrics |

curl http://<CN_IP>:8040/metrics \ | grep star_cache

Heap Profile |

ADMIN EXECUTE ON <id> 'System.print(HeapProf.getInstance().dump_dot_snapshot())'

If the heap profile shows that the memory is held by

StlAllocator

or

ChunkAllocator

but the

mem_tracker

doesn't show it, it confirms a bug in the StarRocks tracking logic for your specific workload. In such cases, upgrading to a later patch (like v3.3.21+) or v3.4 is usually recommended as several negative-tracker bugs were fixed in later releases. References • administration/management/resource_management/Memory_management.md