@Rocky In K8S deployment with 4.1.0 I have the FE pods running but the BE pods are crashlooping and our ops team have a crashhandler reporting this trace below. This runs on x86 and the /proc/cpuinfo on our nodes do report AVX2 support. Is this a known issue ?

Thread 1 (LWP 27):
#0  0x0000000014b4d4f5 in google::LogMessage::Init(char const*, int, google::LogSeverity, void (google::LogMessage::*)()) ()
#1  0x0000000010c3460a in starrocks::CpuInfo::_init_num_cores_with_cgroup() ()
#2  0x0000000010c37521 in starrocks::CpuInfo::init() ()
#3  0x0000000010d6d549 in starrocks::select_hash_functions() ()
#4  0x00007f43a4748304 in __libc_start_main () from /proc/24/root/lib/x86_64-linux-gnu/libc.so.6
#5  0x000000000a5f4025 in _start ()

Need to @ Rocky mate

This is a known issue recently identified in StarRocks version 4.1.0. Based on the stack trace you provided, the crash is caused by a static initialization order conflict. Here is the breakdown of what is happening: Root Cause 1. Static Initialization: When the BE binary is loaded, it runs "static initializers" before the

main()

function starts. One of these initializers is

starrocks::select_hash_functions()

, which is responsible for picking the most efficient hash function (e.g., using AVX2) based on your CPU. 2. Premature CpuInfo Init: To make this choice, it calls

starrocks::CpuInfo::init()

. 3. Glog Initialization Failure: In version 4.1.0, the

CpuInfo

initialization path (specifically

_init_num_cores_with_cgroup

which is triggered in Kubernetes/container environments) contains logging or assertions (

LOG

or

DCHECK

) that depend on the

glog

library. 4. The Crash: Because

main()

hasn't started yet,

google::InitGoogleLogging()

has not been called. When the code attempts to log information about your cgroups/CPU cores,

glog

encounters an uninitialized state and triggers a segmentation fault (

google::LogMessage::Init

). Is this a known issue? Yes. This is currently tracked as a critical bug in the 4.1.0 release (e.g., Issue #71731). It specifically affects 4.1.0 deployments in containerized environments (like K8S) where cgroup detection is performed during startup. Recommended Actions Because this crash occurs before the BE reads its configuration file (

be.conf

), setting parameters like

num_cores

in the config will not prevent the crash. 1. Downgrade: If this is a production or blocker environment, it is recommended to use a stable version from the 4.0.x series (e.g., 4.0.6 or later) until a hotfix (likely 4.1.1) is released. _ _Note: As per the 4.1.0 release notes, if you have already successfully started 4.1.0 and written data, do not downgrade to a version earlier than 4.0.6 due to metadata compatibility changes.* 2. Monitor for Patches: Keep an eye on the StarRocks release notes for version 4.1.1. This issue is a high priority and will be fixed by moving the CPU initialization logic or removing the early log calls. 3. Experimental Workaround: Some users have found that ensuring the container has unlimited CPU resources (no

limits

in K8S) might bypass some of the cgroup parsing logic, but this is not guaranteed to work in all environments and is not a recommended long-term fix. If you are building from source, a temporary fix is to remove the

LOG

or

VLOG

calls inside

src/util/cpu_info.cpp

within the

_init_num_cores_with_cgroup

and

_init_numa

functions. References • administration/management/BE_configuration.md