http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> I'm having issues in my CNs, 2 of t...
# questions-and-troubleshooting
g
@Rocky I'm having issues in my CNs, 2 of them are spewing OOM messages, and I'd like help investigating the issue;:
Copy code
cluster-cn-2 cn be/src/storage/lake/rowset_update_state.cpp:177 _prepare_partial_update_states(segment_id, params, need_lock)
cluster-cn-2 cn be/src/storage/lake/update_manager.cpp:276 state.load_segment(local_id, params, base_version, true , false )
cluster-cn-2 cn be/src/storage/lake/txn_log_applier.cpp:230 check_and_recover([&]() { return apply_write_log(log.op_write(), log.txn_id()); }) tablet_id=48069 txn=txn_id: 70249
cluster-cn-2 cn commit_time: 1782213153
cluster-cn-2 cn combined_txn_log: true
cluster-cn-2 cn txn_type: TXN_NORMAL
cluster-cn-2 cn force_publish: false
cluster-cn-2 cn gtid: 428608277595553792
cluster-cn-2 cn W20260623 11:59:16.635987 140339337754176 transactions.cpp:406] Fail to apply txn log : Memory limit exceeded: Memory of process exceed limit. try consume:3670016 Backend: cluster-cn-2.cluster-cn-search.starrocks.svc.cluster.local, Used: 9847755568, Limit: 13915694038. Mem usage has exceed the limit of BE
cluster-cn-2 cn be/src/storage/lake/rowset_update_state.cpp:177 _prepare_partial_update_states(segment_id, params, need_lock)
cluster-cn-2 cn be/src/storage/lake/update_manager.cpp:276 state.load_segment(local_id, params, base_version, true , false )
cluster-cn-2 cn be/src/storage/lake/txn_log_applier.cpp:230 check_and_recover([&]() { return apply_write_log(log.op_write(), log.txn_id()); }) tablet_id=48058 txn=txn_id: 70249
cluster-cn-2 cn commit_time: 1782213153
cluster-cn-2 cn combined_txn_log: true
cluster-cn-2 cn txn_type: TXN_NORMAL
cluster-cn-2 cn force_publish: false
cluster-cn-2 cn gtid: 428608277595553792
cluster-cn-2 cn W20260623 11:59:16.636845 140346646324800 transactions.cpp:406] Fail to apply txn log : Memory limit exceeded: Memory of process exceed limit. try consume:3670016 Backend: cluster-cn-2.cluster-cn-search.starrocks.svc.cluster.local, Used: 7206928904, Limit: 13915694038. Mem usage has exceed the limit of BE
cluster-cn-2 cn be/src/storage/lake/rowset_update_state.cpp:177 _prepare_partial_update_states(segment_id, params, need_lock)
cluster-cn-2 cn be/src/storage/lake/update_manager.cpp:276 state.load_segment(local_id, params, base_version, true , false )
cluster-cn-2 cn be/src/storage/lake/txn_log_applier.cpp:230 check_and_recover([&]() { return apply_write_log(log.op_write(), log.txn_id()); }) tablet_id=48067 txn=txn_id: 70249
cluster-cn-2 cn commit_time: 1782213153
cluster-cn-2 cn combined_txn_log: true
cluster-cn-2 cn txn_type: TXN_NORMAL
cluster-cn-2 cn force_publish: false
cluster-cn-2 cn gtid: 428608277595553792
cluster-cn-2 cn owsetUpdateStateParams const&, bool)
They are running in pods where the mem limit is 16G. I have 3 of them (cn-0, 1 and 2) and two of them have this same message.
r
Based on the logs provided, your StarRocks Compute Nodes (CNs) are experiencing Out-of-Memory (OOM) errors during the Apply phase of a transaction in a Primary Key table (Shared-data/Lake architecture). The specific failure occurs in 
_prepare_partial_update_states
within the 
lake
storage engine. This happens when the CN attempts to load the metadata and update states required to process a partial update. Analysis of the Error * Memory Limit: Your process limit is ~13.9 GB (
Limit: 13915694038
), which is roughly 90% of your 16 GB pod limit (StarRocks' default 
mem_limit
). * Trigger: The system is failing to allocate even small amounts of memory (e.g., 
3.6 MB
) because the total process memory has reached its limit. * Component: The stack trace points to 
rowset_update_state.cpp
and 
update_manager.cpp
. In the shared-data architecture, partial updates require loading rowset metadata into memory to reconcile which rows are being modified. If you have many concurrent updates or very large rowsets, this memory usage can spike. Recommended Steps for Investigation & Resolution 1. Check Memory Trackers To see exactly what is consuming memory, run the following command from a machine that can reach the CN:
Copy code
bash
curl http://<CN_IP>:8040/mem_tracker
Look for trackers named 
update
, 
index
, or 
RowsetUpdateState
. This will confirm if the Primary Key update states are indeed the primary consumer. 2. Tune Update Memory Limits In StarRocks v3.2+, you can control the fraction of memory reserved for the update manager. * 
update_memory_limit_percent
: (Default 60). This is the percentage of the BE/CN process memory reserved for update-related caches and tasks. * Action: If your CNs are primarily doing updates, you might need to increase this, but since you are hitting the process limit, you may actually need to decrease it to trigger internal eviction earlier or increase the Pod RAM. 3. Adjust Partial Update Configs If you are performing many partial updates, these parameters in 
cn.conf
(or via 
UPDATE information_schema.be_configs
) can help: * 
partial_update_memory_limit_per_worker
: Limits memory used for assembling chunks during partial updates. Reducing this can prevent a single large update from grabbing too much RAM. * 
max_cumulative_compaction_num_singleton_deltas
: If the OOM is also happening during background tasks, lowering this (e.g., to 100) reduces memory pressure during rowset merging. 4. Scale Resources (Recommended) 16 GB is very tight for StarRocks CNs running Primary Key tables, especially with partial updates. * Primary Key tables require memory for the index and the update states. * Recommendation: If possible, increase the pod memory to 32 GB or 64 GB. This provides enough headroom for the OS, the BE process, and the specialized caches used by the Primary Key engine. 5. Verify Table Design If the issue persists even with more memory: * Check if you have a massive number of small rowsets (check 
show tablet <id>
from the log). * Consider if Persistent Index is enabled for these tables to offload some memory to disk (though in Lake mode, this behavior is managed by the `UpdateManager`'s cache). For more detailed logs, you can check the 
<http://be.INFO|be.INFO>
file on the CNs to see the full memory tracker dump that StarRocks usually prints right before an OOM crash. References • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/BE_configuration.md • release_notes/release-2.4.md • release_notes/release-3.5.md
g
@Rocky I checked and the mem usage is solely taken by the "process" section in mem_tracker.
Copy code
level Label Parent Limit Current Consumption Peak Consumption
1	process		12G	10G	13G
2	jemalloc_metadata	process	none	570M	570M
2	query_pool	process	11G	0	533M
2	load	process	3G	0	799M
2	compaction	process	12G	0	1127M
2	schema_change	process	none	0	88M
2	page_cache	process	none	255M	2G
2	jit_cache	process	none	0	0
2	clone	process	none	0	0
2	consistency	process	2G	0	0
2	datacache	process	none	255M	2G
2	replication	process	none	0	0
2	metadata	process	none	1095M	1259M
2	update	process	7G	857M	5G
Can I reduce this particular usage?
r
In StarRocks, the 
process
 tracker represents the total resident set size (RSS) of the BE/CN process as seen by the operating system. When you see a large gap between the 
process
value and the sum of the sub-trackers (like 
query_pool
, 
update
, 
load
), it is usually due to one of three things: memory fragmentation, memory held by the allocator (
jemalloc
) but not yet returned to the OS, or internal caches not explicitly categorized. Based on your output (Current: 10G total vs. ~3.1G in sub-trackers), here is how you can reduce this: 1. Reduce Metadata Cache The 
metadata
tracker is consuming 1.1G. This is likely the Rowset/Tablet metadata cache. In v3.2.10+, you can limit this as a percentage of total process memory. * Action: Add/Update this in 
cn.conf
(or via 
UPDATE information_schema.be_configs
):
Copy code
sql
    -- Default is 30. Try reducing it to 10 or 15 to free up baseline RAM.
    UPDATE information_schema.be_configs SET VALUE = 10 WHERE name = "metadata_cache_memory_limit_percent";
2. Shrink the Jemalloc "Gap" (Memory Release) StarRocks uses 
jemalloc
. It often keeps "dirty pages" (memory it finished using but hasn't given back to the OS) to speed up future allocations. If your 
process
usage is much higher than the sum of trackers, 
jemalloc
is likely holding onto that ~7G. * Investigation: Run 
curl http://<CN_IP>:8040/memz
. Look for "Bytes in page heap freelist" or "extents". If this value is high (several GBs), the memory is "free" but reserved. * Action: You can try to force a release by calling the jemalloc decay (if supported) or by lowering the 
update_memory_limit_percent
. StarRocks doesn't have a direct "ReleaseFreeMemory" HTTP hook for jemalloc like it does for tcmalloc, but you can set 
dirty_decay_ms
to a lower value in your environment variables (
MALLOC_CONF
) before starting the process to make it more aggressive about returning memory. 3. Lower the Update Cache Limit Your 
update
tracker peaked at 5G. This happens during heavy upserts on Primary Key tables. The system might still be reserving a portion of that memory for the `UpdateManager`'s cache. * Action: Reduce the percentage of memory the Update Manager is allowed to occupy.
Copy code
sql
    -- Default is 60%. Reducing this forces the cache to evict Rowset states earlier.
    UPDATE information_schema.be_configs SET VALUE = 40 WHERE name = "update_memory_limit_percent";
4. Check for known "FSLib" memory issues If you are on a version earlier than v3.1.9 or v3.2.4, there is a known issue where the Shared-data "Star Cache" metadata (used for the Data Cache) could consume significant memory that wasn't properly tracked in sub-trackers. * Identification: Check if the directory 
${storage_root_path}/starlet_cache/star_cache/meta
is very large. * Solution: Upgrade to v3.2.4+ or v3.1.9+, or manually clear that directory and restart. Summary of recommended 
cn.conf
adjustments: If you want to tighten memory usage immediately to avoid OOMs on your 16G pods: 1. 
metadata_cache_memory_limit_percent = 10
2. 
update_memory_limit_percent = 40
3. 
max_cumulative_compaction_num_singleton_deltas = 100
(Reduces memory spikes during background merges). Note: If the 
process
usage continues to climb while all sub-trackers remain low, it strongly suggests a memory leak or fragmentation within the allocator, and increasing the Pod RAM to 32G is the most stable long-term fix for production fix. References • administration/management/resource_management/Memory_management.md • knowledge/trouble_shooting/fe_memory_problems.md • administration/management/BE_configuration.md
Open in Slack
PreviousNext
Powered by