@Rocky StarRocks BE/CN ARM64 binary segfaults under cgroup v2 memory limits We're trying to run StarRocks on AWS EKS with ARM64 Graviton instances (r8g.8xlarge, m7g.4xlarge). The BE/CN binary crashes with SIGSEGV (exit 139) during initialization whenever a cgroup v2 memory limit is set on the container. Tested versions:

4.0.8

and

4.1.0-rc01

(both ARM64 builds from Docker Hub) Reproduction: - Pod with no memory limit: starrocks_be starts fine - Pod with any memory limit (16G, 32G, 64G, 128G): segfault on startup - Kubernetes sets memory limits via cgroup v2 (/sys/fs/cgroup/memory.max) - Tested on both Graviton 3 (m7g) and Graviton 4 (r8g) -- same result - Fresh PVCs, no stale data -- still crashes Environment: - EKS 1.34, Amazon Linux 2023, containerd, cgroup v2 -

starrocks/cn-ubuntu:4.1.0-rc01

and

starrocks/cn-ubuntu:4.0.8

(linux/arm64) - Shared_data mode (CN only, no BE) Separate issue:

4.1.0-rc01

also segfaults on x86 AMD EPYC Milan (c6a instances) which support AVX2 but not AVX-512.

4.0.6

and

4.0.8

work fine on the same instances. Was an AVX-512 dependency introduced in 4.1.0? Any guidance appreciated.

This behavior suggests two distinct issues related to hardware-specific instruction sets and system resource detection in recent StarRocks versions. 1. ARM64 Segfault with cgroup v2 Memory Limits The crash (SIGSEGV/exit 139) on Graviton instances specifically when cgroup v2 limits are applied is likely caused by a bug in the BE's memory auto-detection logic. * The Cause: In cgroup v2 environments, the memory limit is read from

/sys/fs/cgroup/memory.max

. If the limit is not set, this file contains the string

"max"

. When a limit is set (e.g., 16G), it contains a numeric value. StarRocks BE/CN attempts to parse this file during initialization. On certain ARM64 kernel/libc combinations (like Amazon Linux 2023 on Graviton 4), the parsing logic or the way it handles the SVE (Scalable Vector Extension) registers during this early check may trigger a null pointer dereference or an out-of-bounds access. * The Fix/Workaround: Bypass the auto-detection by setting a hard memory limit in your

be.conf

(or via the Helm chart's configuration). Instead of leaving

mem_limit

at the default

90%

, set it to a fixed value that matches or is slightly less than your Kubernetes container limit:

bash
    # In be.conf or CN configuration
    mem_limit = 14G  # If your K8s limit is 16Gi

This forces the BE to use the provided value and skip the logic that crawls the cgroup filesystem to determine host/container memory. 2. 4.1.0-rc01 Segfault on x86 (AVX2-only) The crash on AMD EPYC Milan (c6a) instances for version

4.1.0-rc01

while

4.0.8

works confirms a regression in the instruction set compatibility of the 4.1.x release candidate binaries. * The Cause: StarRocks is highly optimized for SIMD instructions. It appears the

4.1.0-rc01

binaries were compiled with AVX-512 instructions enabled by default in the build pipeline. AMD Milan (Zen 3) supports AVX2 but lacks AVX-512 (which was introduced in Zen 4/Genoa). When the BE encounters an AVX-512 instruction it doesn't support, it triggers a

SIGILL

(Illegal Instruction), which often manifests as a

SIGSEGV

or exit 139 depending on how the signal is caught. * The Fix: * Stay on 4.0.x: For hardware that only supports AVX2 (like

c6a

or

m6a

instances), continue using the

4.0.x

stable branch. * Report as Regression: Since

4.0.x

works fine on AVX2, the

4.1.x

branch is expected to maintain that compatibility. You should report this to the StarRocks community or on GitHub. It is likely the RC build was inadvertently built with

-march=skylake-avx512

or a similar flag. * Upgrade Hardware: If you must use

4.1.0-rc

, moving to

c7a

(AMD Genoa) or

c6i

(Intel Ice Lake/Sapphire Rapids) instances which support AVX-512 would resolve the crash. Summary Checklist: 1. For ARM64: Manually define

mem_limit

in the configuration to bypass cgroup v2 auto-parsing. 2. For x86: Revert to

4.0.8

for AVX2-only CPUs or switch to AVX-512 capable instances for

4.1.0-rc01

. 3. Debug Tip: To confirm the AVX-512 issue, run

dmesg -T

on the K8s node immediately after a crash. You will likely see a log entry like

traps: starrocks_be[...] trap invalid opcode

. References • administration/management/BE_configuration.md