@Rocky I have deployed StarRocks 4.1.4 using operator helm with 3 fe replicas with resources 8 core 28 GB. The problem is during peak hours I see a lot of concurrent queries which are queued, as FE handles all the queuing I hit FE CPU limits resulting into connection and query failures. Now FE supports observer nodes beyond 3 replicas but I don't want to keep a lot of nodes running for the entire day since this peak workload is only for 2-3 hours, running for the whole day will add up unnecessary cost. Operator helm does not support FE autoscaling as well, is it safe if I manually add an HPA on my FE pods which can scale from min 3 to max 7 pods so that I will always have 1 leader, 2 follower and rest observer nodes when needed else removed. Can I do this, or will I face issues if I keep adding and removing observer pods dynamically.

Hello! I'm Rocky. Dealing with FE CPU limits during peak concurrency is a common challenge, especially since the FE is responsible for parsing, planning, and managing the query queue. Regarding your plan to use an HPA to dynamically scale FE observer nodes: 1. Can you do it? (Technical Compatibility) * Operator Conflict: You should not manually add an HPA to the FE StatefulSet or Deployment created by the StarRocks Operator. The StarRocks Operator continuously monitors the

StarRocksCluster

(SRC) custom resource. If an HPA modifies the replica count on the StatefulSet directly, the Operator will likely see this as a drift and attempt to scale it back to the value defined in your SRC YAML, leading to a "fight" between the HPA and the Operator. * Lack of Native Support: Unlike Compute Nodes (CN), which have a built-in

autoScalingPolicy

in the StarRocks Operator, the FE component does not currently support native HPA or autoscaling through the Operator. 2. Is it safe to add/remove FE Observers dynamically? While adding observers is a standard way to scale read performance and connection handling, doing it dynamically and frequently (e.g., every 2 hours) carries risks: * Metadata Sync Overhead: Every time a new FE Observer starts, it must synchronize metadata from the Leader/Follower nodes (using BDB JE). For large clusters with significant metadata, this sync process consumes network bandwidth and CPU on your Leader FE, which is already under pressure during peak hours. * Manual Cleanup Required: When the HPA scales the FE replicas down, the Operator will stop the pods, but it does not automatically run

ALTER SYSTEM DROP OBSERVER

. * The dead IPs will remain in the StarRocks metadata. * The Leader FE will continue to attempt heartbeats and metadata synchronization to these "dead" nodes, which adds unnecessary overhead to the Leader's CPU—exactly what you are trying to avoid. * Connection Stability: If your Load Balancer or client application is connected to an Observer that gets terminated by the HPA, those active connections will fail. Recommendations Instead of dynamic FE scaling, consider these more stable approaches: 1. Vertical Scaling: Since you only have 3 FEs, increasing the CPU resources (e.g., from 8 to 16 cores) for those 3 replicas is the most stable fix. This avoids the metadata sync overhead of additional nodes. 2. Scheduled Scaling (Manual/Scripted): If your peak hours are predictable (e.g., 2 PM to 5 PM), you can use a cron job to patch the

StarRocksCluster

CRD to increase

replicas

to 7 before the peak starts and scale back down afterward. Crucially, your script must include a step to log into the Leader and run

ALTER SYSTEM DROP OBSERVER

for each removed node after scaling down. 3. Tuning Query Queuing: If the CPU is spiking specifically due to the queue, check your

max_concurrency_ctrl_queries

and

concurrency_manager_type

settings. Sometimes, lowering the concurrency limit slightly can prevent the FE from over-extending its CPU on planning/parsing too many queries at once. 4. Offload to CN: If the bottleneck is actually the compute cost of the queries and not just the FE's management overhead, consider using Compute Nodes (CN) for your workload. CNs do support native HPA and are designed to scale up and down dynamically without the metadata overhead that FEs have. Summary: I would advise against using a standard HPA on FE pods. If you must scale, use a scheduled approach with a script that handles the

ALTER SYSTEM DROP

commands to keep your metadata clean. References • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • deployment/sr_operator.md

@Rocky Now let's say I keep it constant then what is more beneficial for me in your opinion 16 core 56 GB * 3 or 8 core 28 GB * 5 nodes for FE, so basically whether vertical scaling is more beneficial or horizontal??

For your scenario—specifically dealing with high query concurrency and global query queuing—*vertical scaling (3 nodes of 16 cores / 56 GB) is significantly more beneficial* than horizontal scaling (5 nodes of 8 cores / 28 GB). Here is the technical reasoning why larger nodes are better for StarRocks FEs: 1. The Leader Bottleneck (v3.1.4 and above) In StarRocks v3.1.4+, all running queries are managed collectively by the Leader FE. Every time a query starts or finishes on any FE (Follower or Observer), that node must notify the Leader to update the global concurrency count. * Vertical Scaling: Giving the Leader 16 cores provides more headroom for it to process these high-frequency notifications and manage the query queue without hitting a CPU ceiling. * Horizontal Scaling: Adding more nodes actually increases the burden on the Leader. More nodes mean more network connections, heartbeats, and metadata synchronization traffic (BDB JE) for the Leader to manage, which can exacerbate CPU exhaustion during peak hours. 2. Query Planning Efficiency Query planning (parsing, analyzing, and optimizing) is a CPU-intensive task performed by the FE node the client connects to. * 8 Cores is considered the minimum for production. During peak concurrency, 8 cores can quickly be saturated just by the JVM's Garbage Collection (GC) and metadata background tasks, leaving little room for complex query planning. * 16 Cores allows for more concurrent planning threads and, more importantly, ensures that background metadata tasks don't starve the query planner. 3. Metadata Stability StarRocks stores all metadata in the FE's memory. * Vertical Scaling: Increasing to 56 GB RAM provides a larger buffer for metadata and significantly reduces the risk of Full GC pauses. In StarRocks, if a Leader FE experiences a long GC pause (e.g., > 30s), it may be perceived as dead by the other nodes, triggering a leader election. Frequent leader elections during peak hours will cause massive query failures. * Horizontal Scaling: Smaller memory (28 GB) increases the frequency of GC cycles. Having 5 nodes doesn't help if they are all struggling with memory pressure simultaneously. Summary Recommendation Metric | 3 Nodes (16c/56GB) | 5 Nodes (8c/28GB) | Winner Queue Management | High headroom for Leader | High overhead for Leader | Vertical Metadata Sync | Efficient (fewer peers) | Complex (more peers) | Vertical Query Planning | Faster per-node | More total capacity | Tie Stability | Lower risk of Full GC | Higher risk of Full GC | Vertical My Advice: Upgrade your 3 existing replicas to 16 cores / 56 GB. This provides a more stable foundation for the Leader node to handle the "global queuing" logic. If you still see CPU spikes after vertical scaling, ensure you have set

enable_query_queue_v2 = true

(default in v4.1+) and consider tuning

query_queue_concurrency_limit

to prevent the FE from even attempting to plan more queries than the system can handle. References • administration/management/resource_management/query_queues.md • deployment/plan_cluster.md • introduction/Architecture.md • faq/Others.md References • administration/management/resource_management/query_queues.md

@Rocky Can I have 1 leader, 2 follower and 1 observer on FE or does observers also has to be in odd numbers??

Yes, you can absolutely have 1 Leader, 2 Followers, and 1 Observer (4 FE nodes total). The "odd number rule" only applies to Follower nodes (the nodes that participate in the election and the quorum for metadata writes). Observer nodes do not have to be in odd numbers. Here is the breakdown of why this works: 1. The Odd Number Rule for Followers StarRocks uses the Raft-based BDB JE protocol for metadata. This requires a majority quorum to elect a Leader and commit metadata changes. * If you have 3 Followers (1 Leader + 2 Followers), you need at least 2 to be alive to function. * If you had 4 Followers, you would need at least 3 to be alive. Adding a 4th Follower actually decreases your fault tolerance (you can still only lose 1 node before dropping below a majority, but you have more nodes that could potentially fail). * Therefore, the Follower count (which includes the Leader) should always be 1, 3, or 5. 2. Observers are "Passive" Observer nodes do not participate in voting or the quorum. They simply pull metadata from the Leader and handle query planning/connection management. * Because they aren't part of the election group, having an even or odd number of Observers has no impact on the stability or election logic of the cluster. * You can have 1, 2, 3, 10, or any number of Observers depending on your query load. Your Target Configuration: 4 Pods * 3 Followers: This maintains high availability (can survive 1 node failure). * 1 Observer: This provides an extra node to handle connections and query planning during your peak hours without increasing the quorum requirement. Note on Operator deployment: Since you are using the Helm operator, if you decide to go with a fixed 4-node setup, you will need to ensure your

StarRocksCluster

spec reflects

replicas: 4

for the FE. The Operator will typically initialize the first 3 as Followers and any subsequent nodes as Observers. References • deployment/plan_cluster.md • administration/management/Scale_up_down.md • administration/Meta_recovery.md References • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/FE_configuration.md • administration/Meta_recovery.md