@Rocky How do I register my follower and observer FE nodes to my cluster? I have their AWS IPs

To register your Follower and Observer FE nodes, you need to perform two main steps: register them via SQL on the existing Leader FE, and then start the processes on the new nodes using the

--helper

flag. 1. Register the nodes (Run on Leader FE) Connect to your Leader FE node (typically via port 9030) using a MySQL client and run the following commands. Replace

<ip>

with the AWS IP of the new node and

9010

with your configured

edit_log_port

(9010 is the default). * To add a Follower:

sql
    ALTER SYSTEM ADD FOLLOWER "<follower_ip>:9010";

* To add an Observer:

sql
    ALTER SYSTEM ADD OBSERVER "<observer_ip>:9010";

2. Start the FE nodes (Run on the New Nodes) On each of the new FE instances, you must start the process by pointing it to the existing Leader FE to synchronize metadata. This

--helper

flag is only required the first time you start the node.

bash
# Run this on the new node's terminal
./fe/bin/start_fe.sh --helper <leader_ip>:9010 --daemon

3. Verify the Status After starting the nodes, you can check if they have successfully joined the cluster by running this SQL command on the Leader:

sql
SHOW PROC '/frontends'\G

Look for the

Alive

column (should be

true

) and the

Role

column (should match

FOLLOWER

or

OBSERVER

). Important Notes: * Security Groups: Ensure that the AWS Security Groups allow traffic between all FE nodes on the following ports:

9010

(Edit Log),

9020

(RPC),

9030

(Query), and

8030

(HTTP). * Priority Networks: If your AWS instances have multiple network interfaces, ensure

priority_networks

is configured in

fe.conf

to match the CIDR of your AWS IPs. * Quorum: For high availability, it is recommended to have exactly 3 Followers (1 Leader + 2 Followers). Observers do not participate in voting and do not affect the quorum. References • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md

@Rocky Follower and observer nodes are mainly for load balancing and failover right? It is nowhere close to being the system's query limitations since most of query tiem is spent on backend compute and scas

You are largely correct in your intuition: for most data-heavy analytics, the BE (Backend) nodes do the "heavy lifting" (scanning disks, performing joins, and aggregating data). However, FE (Frontend) scaling is critical for specific limits that have nothing to do with the size of your data, but rather the shape and frequency of your requests. 1. Failover (Followers) Followers are primarily for High Availability (HA). * StarRocks uses the Raft protocol for metadata consistency. You need a majority (quorum) of Followers to be alive to elect a Leader. * If you only have 1 FE and it dies, your whole cluster is down even if your 100 BEs are perfectly fine. * Recommended: A production cluster should have 3 Followers (1 Leader + 2 Followers) to survive the loss of one FE node. 2. Load Balancing & Concurrency (Observers) While BEs do the compute, the FE is the "brain" that orchestrates every query. Adding Observers helps when your bottleneck is: * Query Planning: For every SQL query, the FE must parse it, analyze it, and run the Cost-Based Optimizer (CBO) to create an execution plan. This is CPU-intensive on the FE. * Connection Limits: Each FE has a limit on the number of concurrent MySQL client connections. * Metadata Locks: In very high-concurrency environments, the FE manages locks for metadata. Observers allow you to scale the "read" side of metadata (querying) without affecting the Raft consensus or the Leader's ability to handle "writes" (like

LOAD

jobs or

CREATE TABLE

). 3. When does FE become the bottleneck? You will notice FE limitations in these specific scenarios: * High QPS (Queries Per Second): If you are running thousands of small, sub-second queries per second, the FE's planning overhead becomes the primary bottleneck. * Extremely Complex SQL: Queries with hundreds of joins or massive

IN

clauses can take hundreds of milliseconds just to plan before a single byte is read from the BE. * External Catalogs: If you are querying large Hive/Iceberg tables, the FE spends significant time fetching partition metadata from the external metastore. Summary * Scale BEs when your queries are slow due to data volume or complex compute. * Scale FEs (Followers) to ensure the cluster doesn't go offline if a node fails. * Scale FEs (Observers) when you have hundreds of concurrent users or high QPS and want to offload the planning/connection overhead from the Leader. References • deployment/plan_cluster.md • introduction/Architecture.md • administration/management/resource_management/Load_balance.md • knowledge/trouble_shooting/fe_deadlock_problems.md References • knowledge/trouble_shooting/fe_deadlock_problems.md