@Rocky we're noticing that our queries are not evenly distributed across FE nodes in the cluster. The leader is a hotspot. How do we tune our FE settings so that requests are evenly load balanced across FE nodes?

The fact that your Leader FE is a hotspot usually indicates that client traffic is not being distributed at the network level, or the Leader is overwhelmed by metadata tasks that only it can perform. StarRocks does not have an internal load balancer for client connections, so you must implement distribution externally. Here are the steps to tune your FE distribution and offload the Leader: 1. Implement External Load Balancing If your applications are pointing directly to the Leader FE's IP address, the Leader will bear the brunt of all SQL parsing, planning, and connection management. * Load Balancer (Recommended): Deploy an external load balancer like HAProxy, Nginx, or F5 in front of your FE nodes. Configure it to use a Round Robin or Least Connections strategy across all FE nodes (Followers and Observers). * JDBC Load Balancing: If using Java, you can use the native MySQL JDBC load-balancing syntax to distribute traffic without a dedicated proxy:

text
    jdbc:mysql:<loadbalance://fe_ip1:9030,fe_ip2:9030,fe_ip3:9030/db_name>

2. Offload Metadata Checkpointing (v3.4.0+) By default, the Leader FE is responsible for metadata checkpointing (merging edit logs into a new image file), which can cause significant CPU and I/O spikes. * Setting: Change

checkpoint_only_on_leader

to

false

in

fe.conf

(or via

ADMIN SET FRONTEND CONFIG

). * Effect: This allows the FE cluster to pick a non-Leader node (preferring those with lower heap usage) to perform checkpointing, significantly reducing the Leader's background load. 3. Add Observer FE Nodes If your read concurrency is high, Follower nodes (which participate in Raft elections) may become strained. * Recommendation: Deploy Observer FE nodes. Observers synchronize metadata but do not participate in elections, making them "cheaper" to scale. * Strategy: Point your load balancer for read-only analytical workloads specifically to the Observer nodes, leaving the Followers/Leader to handle writes and cluster management. 4. Configure Read/Write Separation with ProxySQL For the most granular control, you can use ProxySQL. It can be configured to: * Route all DDL/DML (writes) to the Leader FE hostgroup. * Route all

SELECT

queries (reads) to the Follower/Observer hostgroup. This ensures the Leader is only used for operations that strictly require it. 5. Check "Forward to Leader" Behavior StarRocks has a system variable

forward_to_leader

(default

false

). * When

false

, non-leader FEs handle what they can (like simple

SHOW

commands) from their local metadata. * If you find that many administrative commands are still hitting the leader, ensure this is not being set to

true

globally or in session by mistake, which would force metadata queries to the Leader. 6. Verify FE Connection Pool Settings High connection churn can also cause hotspots. Ensure the following are tuned in `fe.conf`: * `mysql_service_nio_enabled`: Ensure this is

true

(default in modern versions) to use non-blocking I/O for client connections. * `max_mysql_service_task_threads_num`: Increase this (default 4096) if you have thousands of concurrent connections. References • administration/management/FE_configuration.md • faq/operation_maintenance_faq.md • sql-reference/System_variable.md

Thanks @Rocky - i already have a loadbalancer service in front of my stateful set. I deploy starrocks on kubernetes btw. does that change your advice at all?

Since you are running on Kubernetes and already have a

LoadBalancer

service, the "Leader Hotspot" issue usually stems from how Kubernetes handles long-lived MySQL connections or background metadata tasks. Here is the specific advice for a StarRocks Kubernetes environment: 1. Check for Session Affinity By default, K8s Services should distribute traffic evenly (

sessionAffinity: None

). However, if your Service was created with

sessionAffinity: ClientIP

, every request from a single client (like a specific BI tool or application server) will be pinned to one FE node—often the first one that became "Ready" (the Leader). * Action: Run

kubectl get svc <fe-service-name> -o yaml

and ensure

sessionAffinity

is set to

None

. 2. Manage Long-Lived Connections Kubernetes

LoadBalancer

(L4) services only balance new connections. Because MySQL/JDBC connections are persistent: * If your application starts up while only the Leader FE is "Ready," all connections will pin to the Leader. They will stay there even after Follower FEs join the cluster. * Action: If you notice a hotspot after a rollout, try a "graceful" restart of your application pods to force them to re-establish connections across the now-available FE replicas. 3. Offload Metadata Checkpointing (Critical) Even if query traffic is perfectly balanced, the Leader FE performs heavy metadata "checkpointing" (merging edit logs) every few minutes. In K8s, where CPU limits are often tight, this causes the Leader to look much "hotter" than Followers. * Action: Enable distributed checkpointing so that any FE can handle this task. * Configuration: Add this to your

StarRocksCluster

CRD under the

starRocksFeSpec.configMapInfo

or directly in the FE config:

properties
    checkpoint_only_on_leader = false

(Note: This requires StarRocks v3.4.0 or later.) 4. Adjust

externalTrafficPolicy

If your LoadBalancer is sending traffic to a node that doesn't have an FE pod, K8s will route it internally (extra hop). If you use

externalTrafficPolicy: Local

, it preserves the client IP and reduces hops, but can cause imbalance if your FE pods aren't distributed evenly across nodes. * Recommendation: For StarRocks,

externalTrafficPolicy: Cluster

(the default) is generally safer for even distribution unless you have a specific need for the original Client IP. 5. Use JDBC-Level Load Balancing Instead of relying solely on the K8s Service to balance, you can provide the application with the DNS names of all FE pods. The StarRocks Operator creates a headless service (usually named

<cluster>-fe-search

). * Action: Update your JDBC connection string to use the multiple FE pod addresses. This allows the JDBC driver to handle the balancing and failover:

text
    jdbc:mysql:<loadbalance://fe-0.fe-service:9030,fe-1.fe-service:9030,fe-2.fe-service:9030/db>

6. Consider a ProxySQL Sidecar If you have extremely high concurrency, a standard K8s L4 Service is often too "dumb" for MySQL protocol balancing. You can deploy ProxySQL as a sidecar to your application pods or as a separate deployment. ProxySQL is aware of the FE roles and can explicitly route: * Writes/DDL: To the Leader FE. * Reads/SELECTs: Distributed across all Followers/Observers. Summary Checklist for K8s: 1. FE Configuration: Set

checkpoint_only_on_leader = false

. 2. Service Config: Verify

sessionAffinity: None

. 3. Application: Ensure your connection pool doesn't create thousands of connections instantly before the cluster is fully up. 4. Scaling: If read load is the issue, add Observer FEs in your

starRocksFeSpec.replicas

(StarRocks will automatically treat replicas beyond 3 as Observers in many operator versions, but check your specific version docs). References • faq/Deploy_faq.md