@Rocky StarRocks 4.1.3 (shared-data): CN CPU is the bottleneck at ~50 ms CPU per query on a 1.8 MB materialized view — can this be reduced? Setup • StarRocks 4.1.3,shared-data mode, storage volume on S3 • Deployed on EKS with the kube-starrocks operator (v1.11.7) • 3 FE + 3 CN, one FE and one CN co-located per node • Nodes: 3 × AWS`m7i.4xlarge`(16 vCPU / 64 GiB), no local NVMe • FE:`requests.cpu=2`,

-Xmx8g

, memory limit 12 GiB • CN:`requests.cpu=8`, memory limit 36 GiB,

mem_limit=30G

• *No CPU limits*on containers (so containers can burst above requests) • Clients connect through an internal NLB → NodePort → kube-proxy (

externalTrafficPolicy: Cluster

) • All configuration is default except the above; no pipeline/parallelism tuning was applied Data • 10 tenants, each with its own database and an async materialized view`mv_cost_by_day` • Each MV:*95,212 rows / 1.77 MB*(7 months of daily cost aggregates, 38 sub-accounts) • Total across all 10 tenants: ~20 MB • Local data cache is enabled and fully warm (

datacache.enable=true

); disk reads are effectively zero during the test • The MV has no`PARTITION BY`(single partition) Query under test

SELECT ChargeDate, ServiceName, BillingCurrency,
       SUM(EffectiveCostSum) AS effective_cost_sum,
       SUM(BilledCostSum)    AS billed_cost_sum
FROM `tenant_<id>`.mv_cost_by_day
WHERE ChargeDate BETWEEN '2026-08-01' AND '2026-08-31'
  AND SubAccountId IN (<8 values>)
GROUP BY ChargeDate, ServiceName, BillingCurrency
ORDER BY ChargeDate, ServiceName;

Closed-loop load generator: N OS processes, one connection each, each looping "send query → receive result → send next" with no think time. Connections are spread evenly across the 10 tenants (each connects as its own StarRocks user). 60 s warm-up discarded, then a 180 s measurement window. Results (zero failed queries in every step) Concurrencyp50 (ms)p95 (ms)p99 (ms)Queries/sec1018192056720242830837303341448905054728291210013229037369320038877288347230055910001125489 Baseline for the network path:

SELECT 1

at concurrency 300 gives p50 10 ms / p95 17 ms and 30,366 queries/sec, so the NLB + kube-proxy path is not the limiting factor. CPU and memory per container (measured from cgroup

cpu.stat

/

memory.current

, 5 s samples) Concurrency 100 (693 queries/sec) ContainerCPU coresMemoryMemory limitcn-08.942.28 GiB36 GiBcn-111.333.01 GiB36 GiBcn-212.292.32 GiB36 GiB*CN total32.56*fe-01.587.70 GiB12 GiBfe-11.287.74 GiB12 GiBfe-21.627.82 GiB12 GiB*FE total4.48Node total37.04 of 48 vCPU (77%)* Concurrency 300 (489 queries/sec) ContainerCPU coresMemorycn-05.992.00 GiBcn-1*14.60*2.63 GiBcn-26.652.04 GiB*CN total27.24FE total3.21Node total30.46 of 48 vCPU (63%)* What we observe 1. *CN accounts for 88–90% of all CPU.*FE stays at 1.0–1.6 cores per instance regardless of load. 2. *Roughly 47 ms of CN CPU is spent per query*(32.56 cores ÷ 693 queries/sec at concurrency 100). For a scan of a 1.8 MB, 95K-row cached table this seems very high to us. 3. *Throughput peaks at ~900 queries/sec around concurrency 30–50 and then declines*(693 at 100, 472 at 200, 489 at 300), while latency keeps growing. 4. *At concurrency 300, total CN CPU actually drops*(27.24 cores vs 32.56 at concurrency 100) while latency roughly triples. This looks like queuing rather than CPU exhaustion. 5. CN load becomes very unbalanced at high concurrency: at concurrency 300 one CN used 14.60 cores while the other two used 5.99 and 6.65. 6. Memory is not a constraint: CN peaks at ~3 GiB against a 36 GiB limit. 7. A*heavier query is faster at high concurrency*: the same query over a 3-month window (3× the scan) gives p95 224 ms at concurrency 100 and 582 ms at concurrency 300, versus 290 ms and 1000 ms for the 1-month window. This suggests a large fixed per-query cost rather than data-volume-bound work. Questions 1. Is ~47 ms of CN CPU per query reasonable for this shape of query on a fully cached 1.8 MB table, or does it indicate a misconfiguration? 2. Which settings should we look at to reduce per-query CPU or raise the concurrency ceiling — for example`pipeline_dop`,

parallel_fragment_exec_instance_num

, query cache, or the number of tablets/buckets (our MV is a single partition and we never tuned bucket count)? 3. Why would total CN CPU decrease while latency increases as concurrency rises from 100 to 300? Is there a queue or thread-pool limit we should raise (or lower)? 4. What causes the CN load imbalance at high concurrency, and how should tablet placement or request routing be adjusted? Note that clients reach FE through an NLB and kube-proxy, so the FE that receives a connection is effectively random. 5. For a workload of many small, high-concurrency aggregations like this, is shared-data mode expected to cost more CPU per query than shared-nothing?

The observation of ~47 ms CPU per query for a 1.8 MB scan in StarRocks 4.1.3 (shared-data) indicates that your workload is dominated by fixed per-query overhead rather than data processing work. 1. Is 47 ms CN CPU reasonable? No, it is high for a 1.8 MB cached scan. In StarRocks, the Pipeline engine and the cost-based optimizer (CBO) are designed for large-scale data. For "tiny" queries, the default settings often cause more time to be spent on scheduling than on execution: * Pipeline Scheduling: Each query is split into pipeline "drivers." If

pipeline_dop

is high (e.g., 8), StarRocks creates 8 threads of execution even for a tiny table, leading to context switching and synchronization overhead. * bRPC Overhead: At high concurrency, the default "single" connection mode for bRPC can become a bottleneck, leading to CPU being spent on network serialization and waiting for IO. 2. Recommended Configuration Adjustments To reduce per-query CPU and raise the concurrency ceiling, apply these settings: * Set `pipeline_dop = 1`: The default (0) auto-adjusts to half your CPU cores (8 in your case). For tiny aggregations, the overhead of parallelism is net-negative.

sql
    SET GLOBAL pipeline_dop = 1; -- Force single-threaded execution per fragment

* Enable Query Cache: This is highly effective for high-concurrency aggregations on small MVs. It caches the intermediate results of the aggregation at the tablet level.

sql
    SET GLOBAL enable_query_cache = true;

* Adjust bRPC Connection Type: To handle high concurrency (QPS > 500), change the bRPC connection mode from

single

to

pooled

in the CN/BE configuration (

be.conf

):

properties
    brpc_connection_type = pooled

* Increase Bucket Count: If your 1.8 MB MV only has 1 bucket (the default for tiny data), only one CN can process a single-partition scan at a time. To distribute the load across all 3 CNs, ensure the MV has at least 3-6 buckets.

sql
    -- Check current bucket count
    SHOW TABLE STATUS FROM tenant_<id> LIKE 'mv_cost_by_day';
    -- Recreate or alter the MV to increase buckets (e.g., DISTRIBUTED BY HASH(...) BUCKETS 6)

3. Why does CPU drop while latency rises? This is a symptom of resource contention and queuing in the communication layer (bRPC) or the FE's internal RPC pool: * When concurrency hits 300, the system may be spending most of its time waiting for a bRPC "stub" or a thread in the

brpc_connection_pool

. * As the workers wait for these resources, actual CPU utilization for data processing drops, but the "time in system" (latency) grows exponentially. * Fix: Increase

brpc_max_connections_per_server

in CN/BE and

brpc_connection_pool_size

in FE. 4. CN Load Imbalance In shared-data mode, data is on S3, but consistency and caching are managed at the tablet level. * If your 10 MVs have 1 bucket each, those 10 tablets are distributed across your 3 CNs (e.g., 4 tablets on one CN, 3 on the others). * Requests for a specific tenant's MV will always be routed to the CN holding the "lease" (leader) for that tablet to ensure cache locality. * Solution: Use more buckets (e.g., 6 or 12 buckets per MV). This ensures that even for a single tenant's query, the scan can be distributed across multiple CNs, evening out the load. 5. Shared-data vs. Shared-nothing * CPU Cost: For a fully cached workload, shared-data mode has a slightly higher CPU cost due to the metadata management of the "LakeService" (checking manifests/versions). * Scale-out: Shared-data is generally better for scaling storage independently, but for absolute minimum latency on "tiny" high-concurrency lookups, shared-nothing (OLAP engine) is slightly more efficient as it bypasses the object storage metadata layer entirely. Summary of Action Plan: 1. Set

SET GLOBAL pipeline_dop = 1;

. 2. Enable

enable_query_cache = true;

.