@Rocky If the FE quorum is broken, is the FE Observer also affected, or does the FE Observer continue to work normally?

When the FE quorum is broken (meaning a majority of Follower FE nodes are down), the FE Observer is significantly affected and the cluster will generally stop providing services. Here is the detailed impact: 1. Loss of Leader Dependency FE Observers do not participate in the quorum (voting), but they rely on a functional Leader to synchronize metadata logs (edit logs). If the quorum is broken, the cluster cannot elect a Leader. Without a Leader, the Observer has no source for metadata updates, and its metadata will become stale. 2. Service Unavailability According to StarRocks documentation, if the majority of Follower nodes are not running, the "FE group will not provide services." While an Observer might still be running as a process and have a local copy of the metadata, it cannot function "normally" for several reasons: * Query Planning: Many query operations and planning tasks require coordination or verification that can only be handled by a Leader or a healthy metadata group. * Cluster Consistency: To prevent split-brain and data inconsistency, the StarRocks FE architecture (based on BDBJE) typically prevents nodes from serving requests if they cannot verify the state of the cluster or find a Master. 3. Recovery Requires Manual Intervention If you lose your FE quorum and only an Observer remains, the Observer cannot automatically take over. To restore service using an Observer node, you must perform a "measure of last resort" recovery: 1. Stop the Observer. 2. Manually change its role from

OBSERVER

to

FOLLOWER

in the

meta/image/ROLE

file. 3. Use the

bdbje_reset_election_group = true

configuration in

fe.conf

to force it to become the new Leader. Summary * Is the Observer affected? Yes. It loses its connection to the metadata master and cannot receive updates. * Does it work normally? No. The FE group as a whole fails to provide service when the quorum is broken. You cannot rely on an Observer to maintain read-only availability if the Leader and its quorum are lost. For more details on handling this scenario, see the StarRocks documentation on Metadata Recovery. References • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • deployment/deploy_manually.md

@Rocky In that case, keeping only 3 FE nodes in the leader election group seems to introduce the risk of a critical outage if multiple FE nodes fail. If we increase the leader election group to 5 FE nodes, the cluster should still be able to maintain quorum even if 2 nodes go down, correct? Also, is it considered a best practice to keep the leader election group at 3 nodes, or do you generally recommend running 5 nodes for higher availability?

Yes, your understanding of the quorum logic is correct, but there are specific trade-offs to consider when choosing between 3 and 5 Follower nodes. 1. Quorum with 5 FE Nodes If you increase the leader election group to 5 FE Follower nodes, the cluster can indeed maintain a quorum if 2 nodes go down. * Quorum Rule: A Leader can only be elected or remain active if more than half of the Followers are alive ($n/2 + 1$). * 3 Followers: Can survive 1 failure (needs 2 alive). * 5 Followers: Can survive 2 failures (needs 3 alive). 2. Is 5 FE Nodes Best Practice? While 5 nodes provide higher fault tolerance, 3 FE Follower nodes is the standard recommendation for most production environments. Why 3 is usually preferred over 5: * Write Performance: Every metadata change (like creating a table or a load job) must be acknowledged by a majority of the election group. Increasing the group from 3 to 5 nodes increases the network round-trips and synchronization overhead for every metadata write operation. * Complexity: Managing 5 nodes increases the likelihood of minor network partitions or individual node issues affecting the election process. * Diminishing Returns: For most users, the probability of 2 out of 3 FEs failing simultaneously is low enough that the performance overhead of a 5-node quorum isn't justified. 3. Recommended Architecture: 3 Followers + Observers If your goal is to increase availability and performance without the overhead of a 5-node election group, the best practice is: 1. Keep 3 Follower FEs: This ensures high availability (survives 1 failure) while keeping metadata synchronization fast. 2. Add Observer FEs: Use Observers to scale your query capacity. Observers do not participate in the quorum, so they don't slow down metadata writes, but they can handle read-only queries and provide additional entry points for users. Summary Recommendation * Use 3 Followers if you want the best balance of write performance and high availability. * Use 5 Followers only if your infrastructure has a high risk of multiple simultaneous node failures and you can tolerate slightly slower metadata operations. * Use Observers to scale out query planning and execution across more machines without affecting the election group's stability. References • deployment/plan_cluster.md • faq/Deploy_faq.md • deployment/deploy_manually.md References • faq/Deploy_faq.md • deployment/plan_cluster.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • knowledge/usage/best-practice.md • administration/Meta_recovery.md