@Rocky I am planning to use AWS NLB with FE to high availability and scalability. Can you tell me the port and address that AWS NLB should use for FE health checks? Even though

/api/health

shows

http 200/Status OK

when there are no active BEs and

total_backend_num=0

(meaning no queries can be processed), I don't think this is sufficient.

For configuring an AWS Network Load Balancer (NLB) to ensure high availability and scalability for StarRocks Frontends (FE), the standard port and address are: * Port:

8030

(Default

http_port

) * Path:

/api/health

* Protocol: HTTP Why

/api/health

returns

200 OK

with 0 Backends You are correct that

/api/health

typically returns an HTTP 200 status as long as the FE process is running and has finished replaying its metadata logs. The response body looks like this:

json
{"online_backend_num":0,"total_backend_num":0,"status":"OK"}

From the FE's perspective, its own service is "Healthy" even if the cluster has no compute resources (Backends). Handling the "Zero Backend" Scenario If you require the NLB to mark an FE as unhealthy specifically when no Backends are available, you have a few options: 1. Custom Sidecar/Health Proxy (Recommended for strict requirements): Since AWS NLB only evaluates the HTTP status code (and cannot parse the JSON body to check if

total_backend_num > 0

), you can deploy a tiny sidecar service (or a cron-based script) on the FE instance. This service would: * Query

<http://localhost:8030/api/health>

. * Parse the JSON. * Return

200 OK

to the NLB only if

total_backend_num > 0

. * Otherwise, return

503 Service Unavailable

. Point your NLB health check to the port of this sidecar. 2. Cluster-Level Monitoring (Standard Approach): In most production deployments,

/api/health

is used as-is. If all Backends (BEs) are down, the entire cluster is essentially down. Marking all FEs as "unhealthy" at the NLB level won't restore service; instead, this is typically handled via external monitoring/alerting (e.g., Prometheus/Grafana) which triggers an incident when

starrocks_fe_backend_num

is zero. 3. TCP Health Check (Less Granular): You can perform a TCP health check on port

9030

(the MySQL

query_port

). This ensures the FE is ready to accept SQL connections, but like the HTTP check, it does not verify the state of the Backends. Important Considerations for NLB & FE * Leader vs. Follower: If you plan to send write requests (DDL/DML) through the NLB, note that all FEs can receive the request, but only the Leader executes it (Followers will forward it to the Leader internally). If you want the NLB to only route to the Leader, you would need a custom script that checks the FE's role (via

SHOW FRONTENDS

) and returns 200 only for the Leader. * Graceful Exit: Starting from StarRocks v3.3, if you perform a graceful shutdown of an FE (

stop_fe.sh -g

), the

/api/health

endpoint will immediately begin returning HTTP 500, allowing the NLB to drain connections before the process terminates. References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md

@Rocky Will FE nodes that are not yet part of the FE cluster also return 200 in /api/health? In this case, I am concerned that the NLB will distribute traffic to FEs that are not yet part of the cluster when they are added.

Yes, your concern is valid. By default, the

/api/health

endpoint on a StarRocks FE returns

200 OK

once the FE process has successfully started its internal HTTP server and initialized its base services. This can happen before the FE has finished replaying metadata or before it has actually joined the BDBJE (Berkeley DB Java Edition) replication group as a FOLLOWER or OBSERVER. Why

/api/health

may not be enough for your scenario 1. Node in "Init" State: When you start a new FE with the

--helper

flag, it enters an initialization phase. While it is waiting for an administrator to run

ALTER SYSTEM ADD FOLLOWER "..."

, the HTTP server is often already active and responding to

/api/health

with a 200 status, even though it cannot yet process SQL queries. 2. No Role Assigned: A node that has not joined the cluster has no role (it is not a Leader, Follower, or Observer).

/api/health

primarily checks if the FE process is "Alive" rather than its specific "Cluster Status." 3. Metadata Replay: An FE that is significantly behind in replaying the edit logs (journal) may still respond with

200 OK

to a basic health check, even if it's not ready to provide up-to-date query results. Recommended Solutions 1. Use the MySQL Port (

9030

) for TCP Health Checks Instead of (or in addition to) the HTTP check, configure your AWS NLB to perform a TCP health check on port

9030

(the default

query_port

). * The FE only begins listening and accepting connections on the MySQL protocol port after it has successfully replayed its metadata and is ready to accept client queries. * Nodes that are started but not yet added to the cluster (

Join: false

) will generally not accept connections on the query port in a way that satisfies a "Ready" state for traffic. 2. Implement a "Ready" Sidecar (Best for strict HA) If you must use HTTP, the most robust way is to deploy a small script or sidecar that performs a more sophisticated check. This script would return

200 OK

to the NLB only if the following are true: *

curl -s <http://localhost:8030/api/health>

returns

total_backend_num > 0

. * The node's state is

Join: true

and

Alive: true

(this requires calling

/api/show_proc

which requires credentials, or checking local logs). 3. Management via Target Groups For the specific issue of adding new FEs, the safest operational pattern is: 1. Deploy the new FE node. 2. Start the FE process. 3. Run the

ALTER SYSTEM ADD FOLLOWER/OBSERVER

command on the Leader FE. 4. Verify the node is

Alive: true

and

Join: true

via

SHOW FRONTENDS

. 5. Only then register the new node as a target in the AWS NLB Target Group. Summary of Ports Service | Default Port | Recommended Use HTTP Health |

8030

| Liveness (Is the process running?) MySQL Query |

9030

| Readiness (Can it process SQL?) Edit Log |

9010

| Internal only (Do not use for NLB) For your requirements, a TCP health check on port 9030 is the most reliable "out-of-the-box" method to ensure the FE is actually ready to serve traffic before the NLB routes requests to it. References • deployment/deploy_manually.md • faq/Deploy_faq.md • faq/operation_maintenance_faq.md • administration/management/enable_fqdn.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_FRONTENDS.md