Hi @Rocky, how to deploy multi-FE?

Hello! I'm Rocky. Deploying multiple FE (Frontend) nodes is essential for achieving high availability (HA) in StarRocks. To deploy a multi-FE cluster (typically 3 Followers for HA), follow these steps: 1. Start the first FE node (Leader) If you haven't already, start your first FE node as the Leader. • Configure

fe/conf/fe.conf

(ensure

meta_dir

and

priority_networks

are set). • Start it:

./fe/bin/start_fe.sh --daemon

. 2. Register the new FE nodes Connect to the first FE using a MySQL client and run the following command for each new node you want to add:

sql
-- Replace <fe_ip> with the IP of the new FE node
-- Replace <edit_log_port> with the port in fe.conf (default is 9010)
ALTER SYSTEM ADD FOLLOWER "<fe_ip>:<edit_log_port>";

Note: If you only need read-only scale-out without participating in elections, use

ADD OBSERVER

instead. 3. Configure the new FE nodes On each new FE instance: 1. Copy the StarRocks FE deployment files. 2. Create the

meta_dir

specified in your config. 3. Edit `fe/conf/fe.conf`: • Important: Ensure all FEs have the same

http_port

. • Set

priority_networks

to the local IP of this specific node. 4. Start the new FE nodes with the Helper flag When starting a new FE for the first time, you must point it to the existing Leader FE so it can synchronize metadata:

bash
# Replace <leader_ip> and <leader_edit_log_port> with the first FE's details
./fe/bin/start_fe.sh --helper <leader_ip>:<leader_edit_log_port> --daemon

Note: You only need the

--helper

flag the first time you start the node. 5. Verify the status Run this command in your MySQL client to check if the new nodes have joined successfully:

sql
SHOW PROC '/frontends'\G

• Alive: Should be

true

. • Role: Should be

FOLLOWER

(or

LEADER

for the first node). Key Tips: * Quorum: For high availability, always deploy an odd number of Follower nodes (3 is recommended). This allows the cluster to survive the failure of one node. * Same Version: Ensure all FE nodes are running the same StarRocks version. * Time Sync: Ensure all node clocks are synchronized (using NTP) to avoid metadata inconsistencies. Let me know if you need help with a specific part of the setup! References • deployment/deploy_manually.md • administration/stargo.md

Hi @Rocky, how does the leader FE know about the follower FEs? Do all follower FEs know about each other? How?

In StarRocks, the communication and mutual discovery between FE (Frontend) nodes are primarily managed through BDBJE (Berkeley DB Java Edition), which provides the high availability and metadata replication framework. Here is how the Leader and Followers "know" about each other: 1. How the Leader FE knows about Follower FEs The Leader FE tracks Follower FEs through two layers: * Explicit Registration (SQL Level): Before a Follower can join the cluster, it must be explicitly added to the metadata by running a command like

ALTER SYSTEM ADD FOLLOWER "IP:PORT"

. This stores the Follower's identity in the cluster's metadata. * BDBJE Replication Group: Once added via SQL, the Leader maintains a "replication group" (internally often represented in BDBJE as

_jeRepGroupDB

). The Leader uses this group to track which nodes are eligible to vote and receive metadata updates (Edit Logs). * Heartbeats: The FE nodes communicate via the

edit_log_port

(default 9010). The Leader sends heartbeats to all registered Followers and Observers to monitor their status and ensure they are "Alive" in the cluster. 2. Do all Follower FEs know about each other? Yes, they do. All Follower nodes in a StarRocks cluster are aware of each other. * Shared Group Membership: Because all Followers are part of the same BDBJE replication group, they all possess the list of every electable member (other Followers) in the cluster. * Paxos-based Elections: Followers need this information to participate in the election process. If the current Leader fails, the Followers must know who the other eligible candidates are to reach a majority consensus and elect a new Leader. 3. How do they discover each other? The discovery happens through a "Helper" mechanism: * The Helper Flag: When you start a new FE node, you use the

--helper

flag (e.g.,

./start_fe.sh --helper Leader_IP:9010

). * Initial Handshake: The new node contacts the Helper (usually the existing Leader) to get the current cluster topology and the latest metadata. * Metadata Sync: Once the initial handshake is successful, the new node receives the full list of all FE nodes (Leader, Followers, and Observers) from the metadata. From that point on, it is a persistent part of the replication group and can communicate with any other node via the

edit_log_port

. Summary of Communication Component | Responsibility BDBJE | Manages the list of nodes, leader election, and log replication. edit_log_port | The physical port (default 9010) used for all FE-to-FE metadata communication.

SHOW PROC '/frontends'

| The command you can use to see all nodes and their roles (Leader, Follower, Observer). In short, they know each other because they all share a synchronized copy of the cluster's membership list, managed by BDBJE and initialized via the helper node. References • administration/Meta_recovery.md

@Rocky does a fe get a new fid on every startup? If the node restarts, does it get a new fid? How does the leader understand of what fid it has get updated?

In StarRocks, an FE (Frontend) node does not get a new identity (often referred to as a "Name" or internal ID) on every startup. Its identity is persistent and tied to its metadata. Here are the specific answers to your questions: 1. Does an FE get a new ID on every startup? No. When an FE node is first started and joined to a cluster, it generates a unique identity. This identity is typically formatted as

IP_Port_Timestamp

(for example:

172.26.92.154_9010_1705568349984

). This information is persisted in the FE's metadata directory (specifically in the

fe/meta/image/ROLE

file). On every subsequent startup, the FE reads this file to identify itself. As long as the

meta_dir

remains intact, the FE identity remains the same. 2. If the node restarts, does it get a new ID? No. A standard restart uses the existing metadata. The FE will resume with its previously assigned name and role. The only scenarios where an FE would get a "new" identity are: * Metadata is wiped: If you delete the

meta

directory and restart the FE (using the

--helper

flag to join again), it will be treated as a brand-new node and generate a new name with a new timestamp. * IP/Port Change: If the IP or port of the host changes and you haven't configured

priority_networks

or FQDN correctly, the FE might fail to start because the identity in the metadata (

ROLE

file) no longer matches the local environment. 3. How does the Leader understand what ID it has? The Leader FE manages the cluster membership using BDBJE (Berkeley DB Java Edition), which acts as the distributed consensus and metadata store. * Registration: When an FE first joins the cluster (via

ALTER SYSTEM ADD FOLLOWER/OBSERVER

), the Leader records that FE's identity and address in the global metadata. * Persistence: This membership list is part of the "edit logs" and "image" files. All FEs in the cluster eventually synchronize this information. * Heartbeats: After startup, the Leader sends heartbeats to all registered FEs. It identifies them by the address and identity stored in its metadata. * Validation: When a non-leader FE starts up, it connects to the Leader. The Leader verifies the incoming node's identity against the registered list. If the FE attempts to join with a different ID but the same IP/Port (or vice versa), the Leader may reject it or treat it as an error to prevent cluster inconsistency. Summary of identification You can check the current identity of your FEs by running:

sql
SHOW PROC '/frontends';

The Name column shows the persistent unique identifier for each FE node. Even after multiple restarts, this Name will remain unchanged. References • administration/management/FE_configuration.md • deployment/deploy_manually.md • administration/Meta_recovery.md

Hi @Rocky, i tried to use the same http port for the follower FE but hit an error

2026-02-12 16:40:33.500-08:00 ERROR (FE Http Server|139) [HttpServer$HttpServerThread.run():291] Fail to start FE query http server[port: 8030] 
java.net.BindException: Address already in use

Hi there! I'm Rocky. It looks like you're encountering a common networking conflict. The error

java.net.BindException: Address already in use [port: 8030]

means that the StarRocks FE (Front End) process is trying to start its HTTP server on port 8030, but that port is already being used by another process on that machine. Here are the most common causes and how to fix them: 1. Check for an existing FE process It's possible a previous FE process didn't shut down correctly or is already running. You can check for processes using that port with:

bash
# Check what is using port 8030
netstat -tunlp | grep 8030
# OR
lsof -i :8030

If you see an existing StarRocks process, you should stop it using

./bin/stop_fe.sh

or kill it before trying to start the new one. 2. Multiple FEs on the same machine Are you trying to run both the Leader FE and a Follower FE on the same physical or virtual machine? * Standard Deployment: Each FE node should ideally be on its own dedicated machine. In this case, they can all use the default port 8030 because they have different IP addresses. * Same Machine Deployment: If you must run multiple FEs on the same machine (e.g., for local testing), you cannot use the same ports. You must edit the

fe/conf/fe.conf

for the follower and change all conflicting ports:

conf
    http_port = 8031
    rpc_port = 9021
    query_port = 9031
    edit_log_port = 9011

3. Check for other services Other applications (like web servers or other databases) might be using port 8030. If another service is using it, you have two choices: * Stop the other service. * Change the

http_port

in your StarRocks

fe.conf

to a different available port (e.g.,

8035

). 4. Correct

priority_networks

If your machine has multiple network interfaces, ensure you have configured

priority_networks

in

fe.conf

to point to the correct CIDR (e.g.,

192.168.1.0/24

). This helps the FE bind to the correct IP address. Summary of next steps: 1. Run

netstat -tunlp | grep 8030

to see what is holding the port. 2. If it's a "ghost" StarRocks process, kill it. 3. If you are intentionally running two FEs on one box, give the second one a unique set of ports in

fe.conf

. Let me know if you need help checking those logs or configurations! References • faq/Deploy_faq.md • deployment/deploy_manually.md • deployment/upgrade.md

Hi @Rocky should I do

alter system add follower

on leader fe first or run the follower fe node first?

Hello! I'm Rocky. When adding a Follower FE node to your StarRocks cluster, the correct sequence is to run the SQL command on the Leader FE first, and then start the new Follower FE node. Here is the step-by-step process: 1. On the Leader FE (via MySQL client): Execute the following command to inform the cluster about the new node:

sql
    ALTER SYSTEM ADD FOLLOWER "<new_fe_ip>:<edit_log_port>";

_(The default

edit_log_port

is 9010)._ 2. On the new Follower FE server: Start the FE process using the

--helper

flag pointing to the Leader FE:

bash
    ./fe/bin/start_fe.sh --helper <leader_fe_ip>:<leader_edit_log_port> --daemon

Why this order? The Leader needs to have the new node registered in its metadata before the new node attempts to join. When the new Follower starts with the

--helper

flag, it contacts the Leader to synchronize metadata and join the BDBJE (Berkeley DB Java Edition) election group. If the Leader doesn't already "know" about this node via the

ALTER SYSTEM

command, the join process may not complete correctly. Important Note: You only need to use the

--helper

flag the first time you start the new FE node. For subsequent restarts, you can just run

./fe/bin/start_fe.sh --daemon

. References • deployment/deploy_manually.md • administration/Meta_recovery.md

Hi @Rocky, for

ALTER SYSTEM ADD FOLLOWER "<new_fe_ip>:<edit_log_port>";

it assumes the http port rpc port and query port stay the same as leader. How to modify it if i do it on the same machine with different ports for the follower fe

Actually, the

ALTER SYSTEM ADD FOLLOWER

command only requires the IP and the

edit_log_port

of the new FE node. It does not assume that the other ports (HTTP, RPC, Query) are the same as the leader; instead, it uses the

edit_log_port

as the primary identifier for metadata synchronization. If you are running multiple FEs on the same machine, you only need to ensure that the new FE's configuration is distinct and that you register it using its specific

edit_log_port

. Steps to add a Follower FE on the same machine with different ports: 1. Prepare a new FE directory: Copy the FE deployment folder to a new location on the machine to ensure it has its own

conf

,

bin

, and

meta

storage. 2. Modify

fe.conf

for the new FE: Change the following ports in the new FE's

conf/fe.conf

to avoid conflicts with the Leader FE: *

edit_log_port

(e.g.,

9011

instead of

9010

) *

http_port

(e.g.,

8031

instead of

8030

) *

rpc_port

(e.g.,

9021

instead of

9020

) *

query_port

(e.g.,

9031

instead of

9030

) * `meta_dir`: Ensure this points to a unique directory. 3. Execute the SQL command on the Leader: Use the IP and the new

edit_log_port

you just configured:

sql
    ALTER SYSTEM ADD FOLLOWER "<ip_address>:9011";

4. Start the new FE node: Start the follower by pointing to the Leader's IP and its

edit_log_port

as the helper:

bash
    ./bin/start_fe.sh --helper <leader_ip>:9010 --daemon

How it works: * Registration: The

ALTER SYSTEM

command tells the Leader to expect a new follower at that specific IP and

edit_log_port

. * Discovery: When the new FE starts and contacts the helper (Leader), it sends its own configuration details (including its unique HTTP, RPC, and Query ports). * Verification: You can verify the different ports are correctly registered by running:

sql
    SHOW PROC '/frontends'\G

This will show a separate row for each FE with their respective

HttpPort

,

QueryPort

, and

RpcPort

. References • administration/management/FE_configuration.md • deployment/deploy_manually.md

Hi @Rocky, how do all follower FEs know about each other to broadcast the kill query request? Is it follower FE (the kill query lands on) -> follower FE (that owns the query)? Or does it need to go through the leader FE? Isthe propagation of follower FE nodeIDs also via leader FE? Or pure peer-to-peer?

Hello! I'm Rocky, and I can help you with that. In StarRocks, the mechanism for FE nodes to discover each other and handle cross-FE requests like

KILL QUERY

relies on the cluster's synchronized metadata and internal RPC system. Here is the breakdown of how this works: 1. How FE nodes know about each other All FE nodes (Follower and Observer) maintain a consistent view of the cluster's state, including the list of all other FEs. This is achieved through: * Metadata Synchronization: FE nodes use BDBJE (Berkeley DB Java Edition) to replicate metadata. When an FE is added to the cluster (e.g., via

ALTER SYSTEM ADD FOLLOWER

), this information is written to the metadata edit log by the Leader FE. * Replication: All Follower and Observer nodes replay these edit logs to update their local memory with the current list of FE nodes, including their IP addresses and RPC ports. * NodeID Propagation: The propagation of node IDs and cluster membership is not pure peer-to-peer. It is a Leader-driven replication model. The Leader FE coordinates the membership, and all other FEs learn about each other by staying in sync with the Leader's metadata. 2. How the

KILL QUERY

request is propagated When you run a

KILL QUERY <connection_id>

or

KILL <connection_id>

on a Follower FE: 1. Local Check: The FE first checks if the

connection_id

belongs to a session it is currently hosting. 2. Broadcast/Targeted RPC: If the connection is not found locally, the FE sends an internal RPC request to other FE nodes. * It does not necessarily have to go through the Leader FE. The FE that receives the

KILL

command acts as the "initiator" and can communicate directly with other FEs using their

rpc_port

(default 9020) because it already knows their addresses from the metadata. * The initiator broadcasts the kill request to the other FEs in the cluster. When the FE that actually "owns" that connection receives the RPC, it terminates the query or connection. Summary * Discovery: Through metadata synchronization (BDBJE) managed by the Leader. * Propagation: Direct RPC from the FE where the command landed to the other FEs (Peer-to-Peer RPC based on metadata info). * NodeID Info: Managed and propagated via the Leader FE through the replication stream. If you are using StarRocks v3.4 or later, you can also use

SHOW PROC '/global_current_queries'

to see all running queries across the entire cluster, which uses a similar internal cross-FE communication mechanism! References • deployment/deploy_manually.md • administration/Meta_recovery.md