@Rocky Is there any bug in starrocks 4.0.12 which cause fe down without any log but process is up and error com.starrocks.qe.scheduler.dag.jobspec.getscannode is null

Based on the symptoms you've described and a review of the StarRocks 4.0 release notes, here is what is likely occurring in version 4.0.12: 1. "No Log" Behavior (Swallowed Exceptions) A known issue was identified in the StarRocks 4.0 line where exceptions thrown by ThreadPool tasks were swallowed by default. This caused tasks to be marked as "finished" even if they failed with a critical error, preventing the error from being written to

fe.log

. * Fix: This was addressed in v4.0.14 (PR #76863), where exceptions in the thread pool are now properly logged and handled. 2. FE "Down" but Process "Up" When the scheduler or coordinator encounters a fatal

NullPointerException

(NPE) that is swallowed by the thread pool, the FE process may continue to run but become unresponsive to new queries or metadata operations. This is often because the internal state of the

DefaultCoordinator

or

DAG Scheduler

has become corrupted or stuck. 3. Error:

jobspec.getscannode is null

This specific error refers to a

NullPointerException

occurring in the

com.starrocks.qe.scheduler.dag

package. * The

JobSpec

class is responsible for defining the specification of a query or load job. * A

ScanNode

is a fundamental part of the execution plan. If the scheduler attempts to access a scan node from a

JobSpec

that was not properly initialized (or is null), it will trigger this error. * Similar issues have been reported in earlier versions (like v3.4.3) where

queryOptions

or

JobSpec

fields were not correctly set for certain job types (like Routine Load or specific Join queries). Recommended Actions 1. Upgrade to v4.0.14 or later: This version includes the fix for swallowed exceptions and several stability improvements for the FE scheduler. 2. Check FE JVM Heap: Even if logs aren't appearing, verify if the FE is hitting a Full GC cycle. A process that is "up" but unresponsive often indicates JVM memory exhaustion. 3. Capture a Thread Dump: While the FE is in this "stuck" state, run

jstack <pid> > thread_dump.txt

and check for any threads in the

com.starrocks.qe.scheduler.dag

package that are stuck or waiting indefinitely. 4. Verify Routine Load/Materialized Views: If this happens during specific operations, check if any Routine Loads or Materialized View refreshes are active, as these are common triggers for

JobSpec

related scheduler bugs. If the issue persists after upgrading, it is recommended to provide the thread dump to the StarRocks community on Slack or open a GitHub issue, as it may be a new edge case in the 4.0 scheduler. References • release_notes/release-4.0.md

@Rocky How to check memory is enough?

To check if your StarRocks nodes (FE and BE) have enough memory, you need to monitor both system-level availability and process-level usage. 1. Check FE Memory (JVM Heap) The FE runs on a JVM, so the most important factor is the JVM Heap usage. If the heap is full, the FE will experience "Full GC," making it unresponsive (appearing "down") even though the process is still running. * Command Line (

jstat

): Run this to see real-time GC behavior.

bash
    jstat -gcutil <FE_PID> 1000 1000

* What to look for: If the

O

(Old Generation) column is consistently above 80-90%, or if

FGC

(Full GC count) is increasing rapidly, your FE memory is insufficient. * StarRocks Metrics: * Check

FE JVM Heap Stat

in your Grafana dashboard. An alert should be set at 80%. * Recommended Config (

fe.conf

): Ensure

-Xmx

(Max Heap) is set correctly based on your metadata size (tablet count): * < 1 million tablets: 16 GB * 1–2 million tablets: 32 GB * 2–5 million tablets: 64 GB * Off-Heap Memory: If the FE process is being killed by the OS (check

dmesg | grep -i oom

), add

export MALLOC_ARENA_MAX=1

to your

fe.conf

to limit glibc memory overhead. 2. Check BE Memory BE memory is managed by an internal tracker and the

mem_limit

setting in

be.conf

. * Memory Tracker (Browser): Visit

http://<BE_IP>:<BE_HTTP_PORT>/mem_tracker

to see a detailed breakdown of where memory is being used (e.g., queries, metadata, load). * TCMalloc Status: Visit

http://<BE_IP>:<BE_HTTP_PORT>/memz

to see physical memory stats. Check the "Bytes in use by application" line. * Metrics (CLI):

bash
    curl -s http://<BE_IP>:<BE_HTTP_PORT>/metrics | grep 'process_mem_bytes'

* Soft vs. Hard Limits: * By default,

mem_limit

is 90% of total system RAM. * Soft Limit (80% of mem_limit): BE will try to release cache. * Hard Limit (90% of mem_limit): BE will start failing queries to prevent a crash. 3. Check System-Level (OS) If the process is "up" but behaving strangely, check if the OS is starving it of resources: * `free -m`: Check

available

memory (not just

free

). *

top

/ `htop`: Look for the RES (Resident) memory of the FE/BE process. * OOM Killer Logs: If the process restarts or disappears, run:

bash
    sudo grep -i "out of memory" /var/log/messages
    # or
    journalctl -k | grep -i oom

Summary Recommendation In your case (FE process "up" but services "down"), the most likely culprit is JVM Heap exhaustion causing a "Stop-the-World" GC pause. 1. Check

jstat -gcutil

. 2. If the Old Gen is full, increase

-Xmx

in

fe.conf

and restart. 3. If memory seems enough but the error persists, capture a thread dump (

jstack <pid>

) to see if the scheduler is deadlocked. References • administration/management/BE_configuration.md