Hello @Rocky I have been using starrocks version 3.5.11 . I have 9be and 3 fe and i am on kubernetes. When one of my be start to startup, it does not start and it logs some current memory statistic and then it raise an exit signal.i provide the logs below.i got this error every time with same memory statistics. I have increased memory to 150 gb and cpu to 18core but i still when i see ps command, i see 100% cpu utilization . On Pod events, i only get failed prestophook and then restating these steps again an again.at last i get connection refused for this be and it fails to start.I want to know how to troubleshoot and find a solution. Here are the logs: replication(0) jemalloc_active(64833564672) jemalloc_allocated(64146833184) jemalloc_metadata(307273984) jemalloc_rss(65138216960) I20260317 111633.265736 139916425520704 daemon.cpp:140] Current memory statistics: process(64416990648) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64833847296) jemalloc_allocated(64147103968) jemalloc_metadata(307276928) jemalloc_rss(65138503680) I20260317 111648.269161 139916425520704 daemon.cpp:140] Current memory statistics: process(64416993464) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64834093056) jemalloc_allocated(64147343392) jemalloc_metadata(307279616) jemalloc_rss(65138765824) I20260317 111703.272639 139916425520704 daemon.cpp:140] Current memory statistics: process(64416996152) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64834351104) jemalloc_allocated(64147573728) jemalloc_metadata(307282304) jemalloc_rss(65139011584) I20260317 111718.276264 139916425520704 daemon.cpp:140] Current memory statistics: process(64416998840) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64834609152) jemalloc_allocated(64147837728) jemalloc_metadata(307284992) jemalloc_rss(65139273728) I20260317 111733.279751 139916425520704 daemon.cpp:140] Current memory statistics: process(64417001528) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64834859008) jemalloc_allocated(64148080160) jemalloc_metadata(307287680) jemalloc_rss(65139535872) I20260317 111748.283287 139916425520704 daemon.cpp:140] Current memory statistics: process(64417004088) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64835104768) jemalloc_allocated(64148314400) jemalloc_metadata(307290240) jemalloc_rss(65139773440) I20260317 111803.286666 139916425520704 daemon.cpp:140] Current memory statistics: process(64417006648) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64835350528) jemalloc_allocated(64148559200) jemalloc_metadata(307292800) jemalloc_rss(65140023296) I20260317 111818.290951 139916425520704 daemon.cpp:140] Current memory statistics: process(64417009208) query_pool(0) load(0) metadata(157301299) compaction(0) schema_change(0) page_cache(0) update(0) passthrough(0) clone(0) consistency(0) datacache(0) jit(0) replication(0) jemalloc_active(64835596288) jemalloc_allocated(64148804000) jemalloc_metadata(307295360) jemalloc_rss(65140273152) daemon.cpp:295 got signal: Terminated from pid: 525(bash), is going to exit .

Based on the logs and symptoms you described, it appears your StarRocks BE is getting stuck in the tablet metadata loading phase during startup, and is subsequently being terminated by Kubernetes (or the entrypoint script) because it fails its health checks (liveness/readiness probes). Analysis of the Logs 1. 100% CPU Utilization: During startup, StarRocks BE must scan the

storage_root_path

and load the metadata (headers) for every tablet assigned to that node into memory. If you have a large number of tablets or rowsets, this process is extremely CPU-intensive as it parses thousands of files. 2. Memory Statistics: *

jemalloc_active(64833564672)

(~60.4 GB) vs

metadata(157301299)

(~150 MB). * The high

jemalloc_active

compared to low StarRocks-tracked metadata suggests that the memory is allocated but hasn't been fully "handed over" to the internal StarRocks trackers yet, or it's consumed by the large volume of tablet header objects being initialized. * Since your limit is 150 GB and you are at ~65 GB, this is not a memory OOM (Out of Memory) crash. 3. The Termination Signal: *

daemon.cpp:295 got signal: Terminated from pid: 525(bash)

* This indicates an external process (likely the Kubernetes

Kubelet

or the

starrocks-manager

bash script) sent a

SIGTERM

to the BE. * In Kubernetes, this usually happens because the Liveness Probe failed. If the BE takes too long to load metadata, its ports (8040, 9050) are not yet open, causing the probes to fail and K8s to restart the Pod. Recommended Troubleshooting & Solutions 1. Adjust Kubernetes Probes (Primary Solution) The BE is likely healthy but just slow to start due to the volume of data. You need to give it more time to complete the metadata loading before Kubernetes kills it. * Increase `initialDelaySeconds`: Set this to a much higher value (e.g.,

300

or

600

seconds). * Increase `failureThreshold`: Allow more failed attempts before a restart. * Use a Startup Probe: If your K8s version supports it, define a

startupProbe

with a long timeout to handle this initial loading phase, separate from the

livenessProbe

. 2. Verify Metadata Loading in Logs Check the

<http://be.INFO|be.INFO>

log files (usually in

storage/log/be.INFO

). Look for lines similar to:

text
Ixxxx xx:xx:xx.xxxxxx xxxxx tablet_manager.cpp:xxx] loading tablet metadata...
Ixxxx xx:xx:xx.xxxxxx xxxxx tablet_manager.cpp:xxx] finished loading tablet metadata.

If you see a long gap between these logs, it confirms the BE is simply busy loading tablets. 3. Check Tablet/Rowset Count A common cause for slow startup is having too many small tablets or versions (rowsets). * Run

show backends\G

from the FE to see the

TabletNum

for that BE (if it was ever up). * If the number of tablets is in the hundreds of thousands or millions, consider increasing the partition granularity or merging small tables. 4. PreStop Hook Failure The "failed prestophook" occurs because Kubernetes tries to run a script (like

be_prestop.sh

) before killing the pod. These scripts often try to connect to the BE's HTTP port to "decommission" or "checkpoint" it. Since the BE hasn't finished starting, the port is refused, causing the hook to fail. You can safely ignore this once you fix the startup timeout. 5. Configuration Optimization If you have a high-core machine (18 cores), you can try to increase the parallelism of metadata loading (though this is often I/O bound). * Check if your disk I/O is saturated during this 100% CPU period (

iostat -x 1

). * If you have many disks, ensure

storage_root_path

is configured to use all of them to parallelize the load. Summary: Your BE is not crashing; it is being killed for taking too long to start. Increase your Kubernetes Liveness/Startup probe timeouts significantly (e.g., to 10-15 minutes) to allow the 64GB of metadata to finish loading. References