@Rocky I’m looking for best practices around the number of FE nodes in a StarRocks cluster. Currently, we operate a FE node group with 3 nodes. Would increasing it to 5 nodes cause any issues? Also, are there real-world cases where people operate with 5 FE nodes? From the best practice documents I’ve seen, it seems that 3 FE nodes are generally recommended. However, the reason I’m considering expanding from 3 to 5 is that if 2 FE nodes go down simultaneously in a 3-node setup, quorum is lost and the cluster can become unavailable. With 5 FE nodes, availability can still be maintained even if up to 2 nodes are shut down or fail at the same time. Could you recommend best practices for determining the appropriate number of FE nodes?

Increasing the number of FE nodes from 3 to 5 is technically supported, but it is rarely done in practice because 3 nodes are generally sufficient for most high-availability (HA) requirements. Here is a breakdown of the trade-offs and best practices to help you decide. 1. Quorum and Fault Tolerance StarRocks uses a protocol (based on BDB JE) that requires a majority quorum of Follower FE nodes to be alive for the cluster to remain available. * 3 Follower nodes: Quorum is 2. It can tolerate 1 node failure. * 5 Follower nodes: Quorum is 3. It can tolerate 2 node failures simultaneously. Your reasoning is correct: moving to 5 nodes increases the cluster's resilience to simultaneous hardware or network failures. 2. Potential Issues with 5 FE Nodes While 5 nodes provide higher fault tolerance, they introduce a few downsides: * Increased Metadata Write Latency: Every metadata change (e.g., DDL statements like

CREATE TABLE

,

ALTER TABLE

, or even the internal commits for data loading) must be acknowledged by a majority of the Follower nodes. With 5 nodes, the Leader must wait for more network acknowledgments, which can slightly increase the latency of these operations. * Inter-node Communication: The synchronization of the edit log across more nodes increases the internal network traffic among FEs, though this is usually negligible unless your cluster has a very high frequency of schema changes or small-batch loads. * Complexity: Managing more stateful nodes increases the maintenance overhead. 3. Best Practices for FE Node Count For most production environments, the recommended path is: * Standard HA (3 Followers): Deploy 3 Follower nodes. This is the "sweet spot" for metadata reliability and performance. * High Read Concurrency (Observers): If your goal is to handle thousands of concurrent query requests rather than improving fault tolerance, do not add more Followers. Instead, add Observer FE nodes. * Followers: Participate in voting and the write quorum. * Observers: Synchronize logs asynchronously and only handle read requests. They do not increase the write quorum, so they don't slow down the leader. * The "Odd Number" Rule: Always maintain an odd number of Follower nodes (1, 3, or 5) to avoid split-brain scenarios and maximize the fault tolerance for the number of nodes used. 4. Real-World Usage * 3 nodes: Used by >90% of StarRocks users. It protects against a single rack or node failure, which is the most common failure mode. * 5 nodes: Typically only used in extreme scenarios, such as multi-region deployments or very large clusters where the cost of a full cluster restart (due to losing quorum) is exceptionally high and the infrastructure allows for two independent failures. Summary Recommendation * Stick with 3 nodes if your primary concern is standard high availability. Double-node failures are rare enough that the performance and simplicity of 3 nodes usually outweigh the benefits of 5. * Upgrade to 5 nodes only if your business requirements strictly mandate survival during a 2-node simultaneous outage (e.g., a "two-rack failure" scenario). * Use Observers if you are seeing high CPU usage on your 3 FE nodes due to query traffic. References • deployment/deploy_manually.md • faq/Deploy_faq.md • deployment/plan_cluster.md