Hi Everyone, We are running StarRocks 3.2.x in production and facing a critical stability issue — both our BE (beservice) and FE (feservice) are getting killed by SIGKILL unexpectedly and repeatedly, causing unplanned restarts multiple times a day. BE is the primary offende We are managing services via Supervisord. While the auto-restart recovers the process within ~5 seconds, this is not acceptable in a production environment — we need to eliminate the root cause entirely. --- Crash Timeline (from Supervisord logs) Jan 4 — FE crashes twice: •

12:39 UTC

→

feservice killed by SIGKILL → restarted → recovered in ~5s

•

20:08 UTC

→

feservice killed by SIGKILL again → restarted → recovered in ~5s

Jan 5 — BE crashes twice within 4 minutes: •

06:45 UTC

→

beservice killed by SIGKILL → restarted → recovered in ~3s

•

06:49 UTC

→

beservice killed by SIGKILL *again* (only 4 mins after restart) → recovered in ~5

The back-to-back BE crashes within 4 minutes strongly suggest the underlying pressure (OOM or CPU saturation) was not relieved after the first restart — the process was killed again as soon as it came back up. --- Suspected Root Causes Based on our analysis, we believe the crashes are caused by one or both of the following: 1. Linux OOM Killer — OS forcefully killing the process due to memory exhaustion. SIGKILL without any application-level error is a classic OOM killer signature. 2. CPU Saturation — Sustained high CPU causing the process to become unresponsive or get killed by an external watchdog. We have not yet confirmed via

dmesg

— we are actively checking and will update. --- What We Need Help With* We are looking for production-proven solutions, not just diagnostics. Specifically: 1. OOM Confirmation & Fix - Best way to confirm OOM killer involvement (

dmesg

patterns,

/var/log/syslog

?) - Recommended

mem_limit

and memory tuning parameters in

be.conf

for StarRocks 3.2.x to prevent OOM kills - Should we set

mem_limit

as a percentage or absolute value for BE in production? 2. CPU Pressure Handling - Are there known StarRocks 3.2.x configs to throttle query/load CPU usage and prevent saturation? - Any recommended cgroups or OS-level limits to protect the BE process? 3. BE → FE Crash Correlation - Can repeated BE crashes destabilize the FE as well, or are these independent failures? - Is there a known cascading failure pattern in 3.2.x we should be aware of? 4. Long-term Stability - What monitoring/alerting setup do you recommend to catch memory/CPU pressure before it leads to a SIGKILL? - Any known bugs or patches in 3.2.x related to this behavior? --- Happy to share: •

be.conf

/

fe.conf

(sanitized) •

dmesg

output • BE/FE logs around crash timestamps • System resource metrics (RAM, CPU) This is a production system and we need a reliable fix. Any guidance from the community or StarRocks team is greatly appreciated. 🙏

@Rocky We are running StarRocks 3.2.x in production and facing a critical stability issue — both our BE (beservice) and FE (feservice) are getting killed by SIGKILL unexpectedly and repeatedly, causing unplanned restarts multiple times a day. BE is the primary offende We are managing services via Supervisord. While the auto-restart recovers the process within ~5 seconds, this is not acceptable in a production environment — we need to eliminate the root cause entirely. --- Crash Timeline (from Supervisord logs) Jan 4 — FE crashes twice: •

12:39 UTC

→

feservice killed by SIGKILL → restarted → recovered in ~5s

•

20:08 UTC

→

feservice killed by SIGKILL again → restarted → recovered in ~5s

Jan 5 — BE crashes twice within 4 minutes: •

06:45 UTC

→

beservice killed by SIGKILL → restarted → recovered in ~3s

•

06:49 UTC

→

beservice killed by SIGKILL *again* (only 4 mins after restart) → recovered in ~5

The back-to-back BE crashes within 4 minutes strongly suggest the underlying pressure (OOM or CPU saturation) was not relieved after the first restart — the process was killed again as soon as it came back up. --- Suspected Root Causes Based on our analysis, we believe the crashes are caused by one or both of the following: 1. Linux OOM Killer — OS forcefully killing the process due to memory exhaustion. SIGKILL without any application-level error is a classic OOM killer signature. 2. CPU Saturation — Sustained high CPU causing the process to become unresponsive or get killed by an external watchdog. We have not yet confirmed via

dmesg

— we are actively checking and will update. --- What We Need Help With* We are looking for production-proven solutions, not just diagnostics. Specifically: 1. OOM Confirmation & Fix - Best way to confirm OOM killer involvement (

dmesg

patterns,

/var/log/syslog

?) - Recommended

mem_limit

and memory tuning parameters in

be.conf

for StarRocks 3.2.x to prevent OOM kills - Should we set

mem_limit

as a percentage or absolute value for BE in production? 2. CPU Pressure Handling - Are there known StarRocks 3.2.x configs to throttle query/load CPU usage and prevent saturation? - Any recommended cgroups or OS-level limits to protect the BE process? 3. BE → FE Crash Correlation - Can repeated BE crashes destabilize the FE as well, or are these independent failures? - Is there a known cascading failure pattern in 3.2.x we should be aware of? 4. Long-term Stability - What monitoring/alerting setup do you recommend to catch memory/CPU pressure before it leads to a SIGKILL? - Any known bugs or patches in 3.2.x related to this behavior? --- Happy to share: •

be.conf

/

fe.conf

(sanitized) •

dmesg

output • BE/FE logs around crash timestamps • System resource metrics (RAM, CPU)

The behavior you’re describing—repeated

SIGKILL

events without application-level logs—is a hallmark of the Linux OOM (Out of Memory) Killer. When the kernel identifies that the system is critically low on memory, it selects the process consuming the most RAM (often the StarRocks BE) and terminates it instantly. Since your services restart and immediately crash again, the memory pressure is likely caused by large resident data (metadata or caches) or queued tasks that resume immediately upon startup. 1. OOM Confirmation & Memory Tuning Confirmation To confirm OOM Killer involvement, run the following commands on the affected nodes: * Check Kernel Buffer:

dmesg -T | grep -i "oom-killer"

* Check System Logs:

grep -Ei 'killed process|oom.kill' /var/log/messages

(or

/var/log/syslog

on Ubuntu/Debian). * What to look for: Look for a line like

Out of memory: Kill process [pid] (starrocks_be)

. This will also print the memory state at the time of the kill. BE Memory Tuning (be.conf) In StarRocks 3.2.x, the default

mem_limit

is 90%. In production, if you have other processes (like Supervisord, monitoring agents, or co-located FEs), this is often too aggressive. * Recommendation: Set

mem_limit

to an absolute value (e.g.,

mem_limit = 100G

) rather than a percentage. This provides a hard buffer for the OS. If using a percentage, reduce it to 80% to allow for OS overhead and page cache. * Lower Load Memory: If crashes occur during heavy ingestion, reduce

load_process_max_memory_limit_percent

(default 30%) to 20%. FE Memory Tuning (fe.conf) FE SIGKILLs are often caused by off-heap memory growth or excessive JVM heap pressure. * Off-heap Fix: Add

export MALLOC_ARENA_MAX=1

to your

fe.conf

. This limits the memory fragments created by the glibc allocator, which is a common cause of FE memory "leaks." * Heap Ratio: Ensure

-Xmx

is set to ~70-80% of the available RAM on the node, leaving room for the OS and off-heap metadata. ────────── 2. CPU Pressure Handling While CPU saturation rarely causes a

SIGKILL

directly (unless a watchdog is present), it can make the process unresponsive, leading to cascading failures. * Query Throttling: Use Resource Groups (enabled by default in 3.2). You can create a resource group with a

cpu_weight

to prioritize critical workloads and use

big_query_cpu_second_limit

to automatically kill queries that consume too many CPU cycles. * Pipeline DOP: If the CPU is constantly at 100%, lower the

pipeline_dop

(Degree of Parallelism). The default is typically the number of physical cores; reducing it can prevent CPU thrashing. * Cgroups: While StarRocks doesn't natively manage OS cgroups for you, running the BE inside a cgroup with a defined

cpu.shares

or

memory.limit_in_bytes

can prevent a single BE from bringing down the entire node. ────────── 3. BE → FE Crash Correlation In 3.2.x, these are usually independent in terms of the kill signal, but interdependent in terms of workload: * Cascading Load: If a BE crashes, the FE has to reschedule all its fragments. This spikes FE CPU/Memory for planning and metadata updates. * Metadata Pressure: If you have a massive number of tablets (>1M), the FE’s metadata memory footprint grows. If BEs are unstable, the FE may struggle with "Tablet Report" processing, which is memory-intensive. ────────── 4. Long-term Stability & Monitoring To catch these issues before a crash: 1. Monitor These Endpoints: * BE:

http://<be_ip>:8040/mem_tracker

— Check which module (e.g.,

query_pool

,

load

,

metadata

) is consuming the most memory. * FE:

http://<fe_ip>:8030/api/health

— Monitor the

jvm_heap_size_bytes

metric via Prometheus. 2. Alerting: Set a "Warning" alert at 80% memory usage and a "Critical" alert at 90%.