http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> When Quorum Is Lost, How can I rec...
# questions-and-troubleshooting
a
@Rocky When Quorum Is Lost, How can I recover in this situation?
r
When a quorum is lost in a StarRocks FE cluster (meaning a majority of Follower nodes are down), the remaining FE nodes cannot elect a leader and the cluster will stop providing services. You can identify this by checking for 
InsufficientReplicasException
or repeated election failure messages in the 
fe.log
. To recover from this situation, you must perform a "metadata recovery" to force one node to become the leader and reset the election group. Step 1: Identify the FE Node with the Latest Metadata Before proceeding, determine which FE node has the most up-to-date metadata by checking the 
meta_dir/image
directory and looking for the highest version number or latest timestamp. Step 2: Perform Recovery (Measure of Last Resort) Based on whether you are recovering from a Follower or an Observer, follow the steps below: Option A: Proceed with a Follower Node If one of your Follower nodes is healthy and has the latest data: 1. Add recovery configuration: Edit the 
fe.conf
of that Follower and add:
Copy code
properties
   bdbje_reset_election_group = true
2. Restart the FE node:
Copy code
bash
   ./fe/bin/start_fe.sh --daemon
3. Verify Leader Status: Log in via MySQL client and run 
SHOW FRONTENDS;
. Check that 
IsLeader
is 
true
and 
Alive
is 
true
. 4. Finalize: Once confirmed, remove the 
bdbje_reset_election_group
line from 
fe.conf
and restart the FE node once more to return to normal operation. Option B: Proceed with an Observer Node If only an Observer node has the latest metadata: 1. Change Role: Edit the file 
fe/meta/image/ROLE
and change 
role=OBSERVER
to 
role=FOLLOWER
. 2. Add recovery configuration: Edit 
fe.conf
and add:
Copy code
properties
   bdbje_reset_election_group = true
3. Restart the FE node: Start it up and verify it becomes the 
LEADER
using 
SHOW FRONTENDS;
. 4. Rebuild the Cluster: Since the election group was reset from an Observer, it is recommended to add new Follower nodes to the cluster and then decommission the temporary leader if necessary. Step 3: Recover Remaining FE Nodes Once you have a single healthy Leader running: 1. Clean metadata: On the other (failed) FE nodes, clear their 
meta_dir
directory completely. 2. Re-add to cluster: • On the Leader, run: 
ALTER SYSTEM DROP FOLLOWER "host:port";
and then 
ALTER SYSTEM ADD FOLLOWER "host:port";
for each node. 3. Restart with Helper: Start the failed nodes using the current Leader as a helper:
Copy code
bash
   ./fe/bin/start_fe.sh --helper leader_ip:edit_log_port --daemon
Important Version Note: • For StarRocks v3.1.10+, v3.2.5+, and v3.3+, use: 
bdbje_reset_election_group = true
. • For older versions (e.g., v2.5, early v3.0/v3.1), you may need to use: 
metadata_failure_recovery = true
. Warning: Metadata recovery is a high-risk operation. Always back up your 
meta_dir
before making manual changes. References • administration/Meta_recovery.md
When you lose a quorum in StarRocks (meaning more than half of your Follower FE nodes are down or cannot communicate), the cluster will stop providing services because it cannot elect a Leader. To recover from this situation, you need to force one FE node to become the new Leader and reset the election group. Please back up your 
meta_dir
directory before proceeding. Step 1: Identify the Node with the Most Recent Data Go to the 
meta/journal
directory on each FE node and check the journal files. The node with the largest numeric suffix in its journal files or the latest timestamp is usually the most up-to-date. Step 2: Configure the Leader Recovery 1. Stop all FE nodes. 2. On the FE node you identified as having the latest metadata, edit its 
fe.conf
file. 3. Add the following configuration:
Copy code
properties
    # For StarRocks v3.1.10+, v3.2.5+, or v3.3+
    bdbje_reset_election_group = true
_(Note: If you are on an older version like v2.5 or v3.0, use 
metadata_failure_recovery = true
instead.)_ Step 3: Start the New Leader 1. Start this FE node:
Copy code
bash
    ./bin/start_fe.sh --daemon
2. Check the logs (
fe.log
) to ensure it has started as the 
LEADER
. 3. Log in via MySQL client and run 
SHOW FRONTENDS;
. You should see this node with 
Role: LEADER
and 
IsMaster: true
. Step 4: Cleanup and Restart 1. Once the node is successfully running as Leader, remove the 
bdbje_reset_election_group
line from 
fe.conf
. 2. Restart this FE node normally. 3. Crucial: Do not simply restart the other failed FE nodes. Their metadata is now inconsistent with the new "master" timeline. Step 5: Re-add Other FE Nodes For the remaining FE nodes: 1. On the Leader node, drop the old failed nodes:
Copy code
sql
    ALTER SYSTEM DROP FOLLOWER "node_ip:edit_log_port";
2. On the failed nodes, delete their existing metadata directory (
meta_dir
). 3. On the Leader node, add them back:
Copy code
sql
    ALTER SYSTEM ADD FOLLOWER "node_ip:edit_log_port";
4. Start the failed nodes using the 
--helper
flag pointing to the current Leader:
Copy code
bash
    ./bin/start_fe.sh --helper leader_ip:edit_log_port --daemon
Summary of Recovery Modes * `bdbje_reset_election_group`: Use this when you have lost quorum but the metadata itself is healthy. * `metadata_enable_recovery_mode`: Use this only if the metadata itself is corrupted and you need to bypass consistency checks to salvage data (last resort). Warning: Manual metadata recovery is a high-risk operation. If this is a production environment, it is highly recommended to reach out to the StarRocks community or support before performing these steps. References • administration/Meta_recovery.md
Open in Slack
PreviousNext
Powered by