Hi, I'm new to StarRocks and am trying to set up my IaC in a prod ready manner. Does the kube-starrocks chart support autoscaling for FE nodes, if it doesn't why?

Hi, as far as I know, the StarRocks Operator does not support autoscaling for FE nodes.

Based on the StarRocks documentation, the StarRocks Kubernetes Operator does not support autoscaling for FE (Frontend) nodes. Currently, automatic scaling is only supported for CN (Compute Nodes). For FE and BE nodes, scaling is a manual process that involves updating the

replicas

field in your StarRocks cluster configuration. Why FE doesn't support autoscaling: FE nodes are responsible for managing metadata and cluster state. Scaling FE nodes (especially Followers) involves maintaining a quorum (an odd number of followers is required for High Availability), which makes them unsuitable for the rapid, automated scaling typical of Horizontal Pod Autoscalers (HPA). How to scale FE nodes (Manual): You can scale FE nodes manually by patching the

StarRocksCluster

resource:

bash
kubectl -n starrocks patch starrockscluster <cluster_name> --type='merge' -p '{"spec":{"starRocksFeSpec":{"replicas":3}}}'

For comparison: * CN Nodes: Support

autoScalingPolicy

using HPA (Horizontal Pod Autoscaler) based on CPU or memory utilization. * BE Nodes: Support manual scaling. Similar to FE, they are not typically autoscaled automatically because scaling in requires data redistribution (decommissioning) to ensure data safety. If you are looking to scale your cluster dynamically based on query load, it is highly recommended to use CN (Compute Nodes), as they are designed to be stateless and fully support the Operator's autoscaling features. References • deployment/sr_operator.md • administration/management/Scale_up_down.md • administration/management/FE_configuration.md • quick_start/helm.md

Thanks @ahn heesuk

Also, a single FE node can handle around 4,000 connections, and since FE mainly handles query planning while the actual computation is performed on BE or CN nodes, FE nodes generally do not require a lot of resources. Is there a specific reason or use case where you would want to autoscale FE nodes?

I will be doing accelerated model training using a big amount of historical data and am unsure how many FE pods I should have to handle concurrent requests, thus thought it'd be simpler if have autoscaling

Ah, I see. If it becomes necessary, you could also try setting it up by your selfs. Although the StarRocks Operator doesn’t support this directly, you can create a StatefulSet for FE Observers under

templates/

and register them as observers to the FE Leader. FE Observers are not part of the quorum but can still handle queries. you could then apply an HPA to the observer group.

@Rocky how do I spawn an observer FE node

To spawn an Observer FE node in StarRocks, you need to register it in the cluster via SQL and then start the process on the new machine with a helper flag. Here is the step-by-step process: 1. Register the Observer Node Connect to your existing StarRocks cluster (via the Leader FE) using a MySQL client and run the following command:

sql
-- Replace <fe_address> with the IP of the new node 
-- and <edit_log_port> with its edit_log_port (default is 9010).
ALTER SYSTEM ADD OBSERVER "<fe_address>:9010";

2. Prepare the New Node On the new server where you want to run the Observer FE: 1. Extract the StarRocks package and navigate to the FE directory. 2. Configure

fe/conf/fe.conf

: * Ensure

priority_networks

is set correctly to the node's network (e.g.,

priority_networks = 192.168.1.0/24

). * Ensure the

meta_dir

points to a valid directory for metadata storage. 3. Create the metadata directory:

bash
    mkdir -p /path/to/your/meta_dir

3. Start the Observer Node Start the FE process by pointing it to the existing Leader FE as a helper. This is required for the new node to synchronize the initial metadata.

bash
# Navigate to the FE bin directory
cd fe/bin

# Start the node (replace <leader_ip> and <leader_edit_log_port> with the Leader's info)
./start_fe.sh --helper <leader_ip>:9010 --daemon

Note: You only need the

--helper

flag the first time you start the node. Subsequent restarts do not require it.

4. Verify the Status Back in your MySQL client, check if the new node has joined successfully:

sql
SHOW PROC '/frontends'\G

Look for the row corresponding to your new node's IP. You should see: * Role:

OBSERVER

* Alive:

true

* Join:

true

Why use an Observer? Observers are used to scale out the querying capacity of the FE layer. Unlike Followers, they do not participate in the BDBJE voting process (election), meaning they don't add overhead to metadata writes but can handle many read-only requests. References • deployment/deploy_manually.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md

Thanks, seems simple enough, could try it if needed. It seems like it'd be a good addition to the operator/starrocks chart to have that option out of the box, could contribute it, I assume that's ok?

I agree it would be nice to have it as a default option in the chart. However, I remember there was previously an issue about adding Observer nodes in the StarRocks Operator. I’m not sure what the current status is though. 😅

in short it is not supported for adding observers yet. However, there is a PR contributing this ability, check it out at https://github.com/StarRocks/starrocks-kubernetes-operator/pull/734 and possibly you can also review the PR and provide feedback.

thanks