@Rocky How do you maintain BE

labels.location

with the Kubernetes operator? We’re evaluating two location-labeled BE pools in one cluster, with tables and asynchronous MVs assigned to one pool or the other. StarRocks documents setting a BE’s location through FE SQL:

ALTER SYSTEM MODIFY BACKEND "<host>:9050"
SET ("labels.location" = "pool:specialized");

We deploy with the StarRocks Kubernetes operator. Its

podLabels

appear to set Kubernetes Pod metadata, while StarRocks placement uses the backend

Location

shown by

SHOW BACKENDS

. We also see one BE specification and one BE StatefulSet per cluster. Our operational concern is new or changed FE backend records, rather than every Pod restart. FE persists the location on an existing backend record, but a new record starts with an empty location. If scale-out or a replacement creates a new record while FE is temporarily unavailable to our automation, a one-time SQL step could be missed. We would then have Kubernetes BEs whose FE locations do not match the intended pools. How do other operator users handle this? 1. Is there a supported way to assign a BE location during registration, or to have the operator maintain it afterward? Is an external reconciler the expected approach? 2. Which Pod replacement or BE recovery events retain the existing FE backend record, and which create a new one? Are we overestimating the drift risk? 3. What is the recommended way to run two independently sized BE pools in one operator-managed cluster? 4. For strict separation, how do you handle the documented replica-count fallback when a pool has too few eligible BEs? Can a pool location coexist with an existing rack location? The operator already queries FE and runs SQL for part of the CN lifecycle. Would BE location reconciliation be an appropriate upstream operator feature, or is there an existing pattern we should use?

To maintain StarRocks BE

labels.location

within a Kubernetes environment managed by the operator, you are touching on a specific gap between the declarative nature of Kubernetes and the imperative SQL-based metadata management in StarRocks. Here is a breakdown of how this is typically handled and the specifics of backend persistence: 1. Assigning BE Location during Registration There is currently no supported way to assign a BE location via

be.conf

or a environment variable at startup. StarRocks BEs do not "self-report" their location label to the FE; the label is a piece of FE metadata. * Manual/Automated SQL: The only way to set the label is via the

ALTER SYSTEM MODIFY BACKEND

SQL command. * Operator Support: As you noted, the current StarRocks Kubernetes Operator (v1.x) does not yet have a native

labels.location

field in the

starRocksBeSpec

that it reconciles into the FE. * External Reconciler: Using an external reconciler or a specialized Job/CronJob is the standard pattern for users requiring this level of automation. A common approach is a small script that queries

SHOW BACKENDS

, identifies new pods (using a naming convention or K8s annotations), and runs the necessary

ALTER SYSTEM

command. 2. Pod Replacement and Backend Record Drift The risk of "record drift" (new BE records with empty locations) depends heavily on whether you are using FQDN or IP-based registration: * FQDN (Recommended): The operator typically configures BEs to use FQDNs (e.g.,

be-0.be-service.ns.svc

). Because StatefulSet pod FQDNs are stable across restarts and even rescheduling to new nodes, the FE sees the same "host" and retains the existing backend record, including its

labels.location

. * IP-based: If the cluster is configured to register via IP, a pod replacement that results in a new IP will cause the FE to treat it as a new backend. The old record will eventually be marked

Dead

and the new one will appear with an empty location. * Risk Evaluation: If you use FQDNs, your risk is limited to Scale-out events (adding

be-3

,

be-4

). Pod restarts and node replacements (where the pod retains its identity/PVC) generally do not lose the label. 3. Multiple BE Pools in One Operator Cluster The official StarRocks Operator currently only supports one

starRocksBeSpec

per

StarRocksCluster

CR. To run two independently sized and labeled BE pools, you have two main options: 1. Multiple CRs (Experimental): You can technically deploy a second

StarRocksCluster

CR that only contains a

starRocksBeSpec

, pointing it to the FE service of the first cluster via configuration. However, the operator is not explicitly designed for this "multi-BE-spec" architecture in a single management loop. 2. CN Groups (Shared-Data Mode): If you are using shared-data mode, the operator supports multiple CN (Compute Node) groups. While CNs don't store data, they are the primary compute unit in that architecture and can be grouped/labeled. For shared-nothing (local storage), you are currently limited to a single BE StatefulSet. 4. Replica-count Fallback and Rack Coexistence * Replica Fallback: StarRocks prioritized availability over isolation. If you specify

PROPERTIES ("labels.location" = "pool:A")

but pool A has fewer BEs than the

replication_num

, StarRocks will place the remaining replicas on other available BEs. To ensure strict separation, you must ensure each pool has at least

replication_num

healthy nodes. * Rack Coexistence: A BE can only have one location label value at a time (e.g.,

"labels.location" = "pool:specialized"

). You cannot currently set both a "pool" and a "rack" as separate first-class location labels. However, you can use a composite string like

"pool:specialized,rack:1"

if your automation is designed to parse it, though the FE logic generally treats the entire string as the identifier. Recommended Pattern

Thanks. Could you confirm the supported pattern for two location-labeled BE pools under one operator-managed FE cluster? In particular, is an external SQL reconciler the intended solution, and is there a supported way to manage a second BE StatefulSet? We understand FQDNs preserve the FE record across ordinary Pod replacement, but new records still need

ALTER SYSTEM MODIFY BACKEND

. Also, StarRocks 4.0.14 rejects

pool:specialized,rack:1

as a BE location; is there a supported way to express both pool and rack constraints? Finally, should we treat location placement as best effort whenever eligible capacity drops below the replica count? @Rocky

To address your operational concerns about maintaining BE

labels.location

with the Kubernetes operator, here is the current state of supported patterns and recommended practices. 1. Manual vs. Automated Reconciliation There is currently no native support in the StarRocks Kubernetes Operator (v1.x) to declaratively manage backend labels within the

StarRocksCluster

CR. * The Problem: Backend labels are metadata stored in the FE's image/journal. When a new BE registers (e.g., during scale-out or if using IP-based registration), the FE creates a blank record. The BE cannot "self-report" labels via

be.conf

. * The Solution: An external SQL reconciler is the standard approach used by the community. * Implementation: A lightweight Job or Sidecar container that periodically runs

SHOW BACKENDS

, compares the results against your desired state (often derived from K8s Pod labels or Annotations), and executes

ALTER SYSTEM MODIFY BACKEND ... SET ("labels.location" = "...")

. * Why it's needed: Even with FQDNs, a scale-out event creates a brand-new backend ID in the FE that must be initialized with its pool label before it can correctly host replicas for your specialized MVs. 2. Multi-Pool Management in the Operator The current operator maintains a one-to-one relationship between a

StarRocksCluster

CR and a single BE StatefulSet. * BE Limitation: You cannot currently define two distinct

starRocksBeSpec

groups in one CR. * Alternative for Compute: If your "pools" are primarily for query isolation (rather than data storage isolation), you should use CN (Compute Node) groups. The operator does support multiple

starRocksCnSpec

groups, which can be independently sized and scaled. * Workaround for BEs: For strict storage separation (two BE pools), users sometimes deploy a second "BE-only"

StarRocksCluster

CR that points to the first cluster's FE service. However, this is advanced and requires manual coordination of the FE addresses in the second CR's configuration. 3. Label Syntax and "Rack + Pool" Coexistence StarRocks 4.x (and 3.x) treats the backend

labels.location

as a single string identifier. * Rejection Reason: The FE property analyzer expects a simple

key:value

format for the backend. Commas (

,

) are only supported in Table/MV properties to allow data to span multiple backend locations (e.g.,

PROPERTIES("labels.location" = "rack:1,rack:2")

). * The Pattern: To express both a pool and a rack, you must encode them into a single value, such as

SET ("labels.location" = "pool_specialized_rack_1")

. You then set your Table/MV properties to match that specific concatenated string. 4. Replica Placement Fallback The "best effort" behavior is indeed the intended design: * Availability First: If a pool's BE count drops below the

replication_num

(e.g., you have 3 replicas but only 2 BEs in

pool:specialized

), StarRocks will place the third replica on any available BE in the cluster to maintain data safety. * Strictness: There is currently no "Hard Affinity" toggle that would cause an insert to fail rather than violate the location constraint. To achieve strict separation, you must ensure your specialized pool is over-provisioned or use

storage_volume

(in shared-data mode) for physical isolation at the storage level. Summary of Recommendations * Use FQDNs: Ensure

use_hostname: true

is set (this is the operator default) to keep backend records stable during routine Pod restarts. * Build a Reconciler: Treat BE label assignment as a post-provisioning step. A simple script watching the K8s API for new BE Pods and running the

ALTER SYSTEM

SQL is the most reliable path. * Single-Key Encoding: Use a flat string like

pool-A-rack-1

for your backend location labels to satisfy the FE's parser while capturing multi-dimensional topology. References