@Rocky we keep on seeing the segementation errors on BE and crashing, when we try to upgrade it to 4.1.0, However when we increase the number of cpus to greater then 16, it comesup. how does BE calculating number of cpus .? even though BE doesn't use 16 CPUs. How does BE calculating number of cpus .?

Hello there! I'm Rocky. It sounds like you're running into an interesting (and frustrating) issue with the very latest StarRocks 4.1.0 release candidate. Seeing a segmentation fault that disappears only when you bump the CPU count to 16 is a specific symptom usually tied to how the BE (Backend) initializes its internal parallel processing engines. To answer your question directly: Here is how the StarRocks BE calculates the number of CPUs. 1. How the BE Calculates CPU Count The BE uses a utility class called

CpuInfo

to determine the "effective" number of cores. It follows this priority order: * Manual Override: First, it checks the

num_cores

parameter in your

be.conf

. If this is set to a positive integer (e.g.,

num_cores = 16

), the BE ignores all auto-detection logic and uses that value. * Cgroup Limits (Containers): If you are running in a container (Docker/Kubernetes), the BE reads the cgroup limits to avoid over-subscribing the physical host: * Cgroup v2: It reads

/sys/fs/cgroup/cpu.max

. * Cgroup v1: It calculates the ratio of

/sys/fs/cgroup/cpu/cpu.cfs_quota_us

to

/sys/fs/cgroup/cpu/cpu.cfs_period_us

. * Physical Hardware: If no cgroup limits are found, it parses

/proc/cpuinfo

(specifically

_SC_NPROCESSORS_ONLN

via

sysconf

) to count the active logical cores on the machine. 2. Why it crashes with < 16 CPUs in v4.1.0 StarRocks 4.1.0 introduced several high-concurrency features designed for large-scale data management, including Parallel Compaction and Automatic Tablet Splitting (Multi-Tenant management). The segmentation fault likely occurs because: * Thread Pool Sizing Logic: Many internal thread pools (like

pip_exec

,

pip_scan

, and the

load_channel_rpc_pool

) calculate their size as a factor of the core count. In version 4.1.0, some of these new operators may have a bug where a core count below the "recommended" 16 results in a zero or negative thread pool size, leading to an invalid memory access (segmentation fault) during startup. * Baseline Verification: As of recent versions, StarRocks has added stricter "System Resource Forced Verification." If the core count is detected as too low, the BE might attempt to initialize a "minimal" environment that hasn't been fully tested for the new v4.1 parallel architecture. 3. Recommended Solution Since you've noticed that the BE doesn't actually need all 16 CPUs to run—it just needs the calculation to result in 16—you can bypass this auto-detection bug by manually setting the core count. Modify your

be.conf

on each node:

properties
# Manually override the CPU detection to satisfy the thread-pool sizing logic
num_cores = 16

By setting this, the BE will initialize its internal structures as if it has 16 cores available, which should stop the segmentation fault. You should also ensure that your memory is at least 64GB, as the v4.1 parallel features are quite memory-intensive. If you continue to see crashes after this change, I recommend checking the

be.out

file for the specific stack trace of the segmentation fault! References • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md • faq/Others.md • faq/resource_isolation_faq.md • best_practices/audit_log_resource_group.md • administration/management/resource_management/resource_group.md