http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> Using the Kubernetes Operator for S...
# questions-and-troubleshooting
s
@Rocky Using the Kubernetes Operator for StarRocks, is there a way to have HA? I have 3 replicas set up (3 BE, 3 FE), but as soon as one BE gets killed/node rotated, the whole cluster goes down/quorum is lost. I created a 
PodDisruptionBudget
in 
resources.pdb
in the Helm chart which targets the cluster to only have a single pod restarted at the same time, but to no avail.
r
To achieve true High Availability (HA) with the StarRocks Kubernetes Operator and prevent quorum loss during node rotations or pod restarts, setting 
replicas: 3
is only the first step. The behavior you're seeing—where losing a single node or BE pod brings down the cluster—is typically caused by pod co-location or incorrect replication settings. Here is how to properly configure HA to survive node rotations: 1. Implement Pod Anti-Affinity (Crucial) By default, Kubernetes might schedule multiple Frontend (FE) pods on the same physical node. If that node is rotated, you lose the majority of your FEs, and Raft quorum is lost. You must add 
podAntiAffinity
to your 
StarRocksCluster
CRD for both FEs and BEs to ensure they are distributed across different nodes.
Copy code
yaml
spec:
  starRocksFeSpec:
    replicas: 3
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: <http://app.kubernetes.io/component|app.kubernetes.io/component>
                  operator: In
                  values:
                    - fe
            topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
  starRocksBeSpec:
    replicas: 3
    affinity:
      podAntiAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          - labelSelector:
              matchExpressions:
                - key: <http://app.kubernetes.io/component|app.kubernetes.io/component>
                  operator: In
                  values:
                    - be
            topologyKey: "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
2. Correct PodDisruptionBudget (PDB) If you are using node draining (common in EKS/GKE upgrades), a PDB is required to tell the cluster to only evict one pod at a time. Ensure your PDB labels match the labels applied by the operator (usually `app.kubernetes.io/component`). * FE PDB: Protects the metadata quorum. * BE PDB: Protects data availability. Example PDB for FEs:
Copy code
yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: starrocks-fe-pdb
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      <http://app.kubernetes.io/component|app.kubernetes.io/component>: fe
3. Verify Table Replication "Quorum lost" is specifically an FE/metadata error. However, if your queries are failing when a BE goes down, verify that your tables are using 3 replicas. If a table was created with 
PROPERTIES("replication_num" = "1")
, losing the BE hosting that single replica will cause queries to that table to fail immediately. * Check: Run 
SHOW CREATE TABLE <table_name>;
and look for 
replication_num
. It should be 
3
. 4. Ensure Persistent Storage for FEs FE pods must use persistent volumes to store their metadata and Raft logs. If you are using 
emptyDir
, an FE pod will lose its metadata upon restart and will fail to rejoin the HA group correctly, causing quorum issues. * Ensure 
storageSpec
is configured in your 
starRocksFeSpec
. Summary Checklist 1. 3 FEs and 3 BEs are deployed. 2. Pod Anti-Affinity is set to 
required
so pods are on separate nodes. 3. FE PDB is set to 
maxUnavailable: 1
. 4. Tables are created with 
replication_num: 3
. 5. FE Role: Run 
SHOW PROC '/frontends';
inside StarRocks to ensure you have 1 Leader and 2 Followers (not Observers). If these are in place, you should be able to lose any single node or pod without the cluster going down. References • quick_start/helm.md • administration/management/FE_configuration.md • deployment/sr_operator.md • faq/Deploy_faq.md
s
@Rocky I have the anti affinity, the PDB are set as expected, and the tables aren't created explicitly with 
replication_num: 3
, but the tables are with that parameter when using 
SHOW ...
Copy code
PROPERTIES (
"compression" = "LZ4",
"enable_persistent_index" = "true",
"fast_schema_evolution" = "true",
"partition_live_number" = "15",
"replicated_storage" = "true",
"replication_num" = "3"
);
r
The behavior you are describing—where losing a single BE pod or node leads to a "quorum lost" error—is almost certainly an issue with your Frontend (FE) configuration rather than the BEs. In StarRocks, "Quorum lost" refers specifically to the metadata group (FEs) failing to maintain a majority (n/2 + 1) in their Raft/BDBJE group. BEs do not participate in this quorum; if all BEs died, the FE would still stay up and report that metadata is healthy. Here is why your cluster is likely losing quorum during a node rotation and how to fix it: 1. Verify FE Roles (Follower vs. Observer) High Availability (HA) only works if your FEs are registered as Followers. By default, the first FE is a Leader (a type of Follower), but if the subsequent FEs were added as Observers, they do not count toward the quorum. * Check: Run 
SHOW FRONTENDS;
or 
SHOW PROC '/frontends';
in your SQL client. * Requirement: You must see 3 FEs with 
Role: FOLLOWER
(one will be 
LEADER
). If you see 
Role: OBSERVER
, those nodes cannot participate in leader election, and losing the single Leader will bring the cluster down. 2. FE Persistence is Mandatory If your FEs are not using Persistent Volume Claims (PVCs), they will lose their metadata and "identity" (the 
term
and 
node name
in the BDBJE group) as soon as the pod is killed. * When a new FE pod starts without its old metadata directory, it cannot rejoin the existing Raft group automatically as a "Follower" because it looks like a brand new node. * Fix: Ensure your 
starRocksFeSpec
in the 
StarRocksCluster
CRD includes a 
storageSpec
with a PVC.
Copy code
yaml
    starRocksFeSpec:
      replicas: 3
      storageSpec:
        name: fe-meta
        storageClassName: "your-storage-class"
        size: 10Gi
        mountPath: /opt/starrocks/fe/meta
3. PDB Label Selectors You mentioned creating a 
PodDisruptionBudget
. If you created it manually in 
resources.pdb
, ensure the matchLabels target the FEs specifically. If the PDB only targets BEs, Kubernetes might drain a node containing an FE at the same time another FE is restarting, leading to 2/3 FEs being down (Quorum Lost). * Fix: Ensure you have a PDB specifically for FEs:
Copy code
yaml
    selector:
      matchLabels:
        <http://app.kubernetes.io/component|app.kubernetes.io/component>: fe
4. "Replicated Storage" vs. Availability The property 
"replicated_storage" = "true"
in your table metadata is a StarRocks 3.x feature for "Single Leader Replication." While it improves write performance, it still relies on the BEs being healthy. * If you lose 1 BE and have 
replication_num = 3
, the table remains available. * However, if you see the specific error "Master not found" or "Quorum lost", the BE is a red herring—the actual failure is that the FE leader has vanished and the remaining FEs cannot elect a new one. 5. Termination Grace Period During node rotation, K8s sends a 
SIGTERM
. If the StarRocks FE doesn't shut down and hand over its "Leader" role within the grace period (default 30s), the remaining FEs might take longer to realize the leader is gone, leading to a brief "service unavailable" window. * Fix: Set 
terminationGracePeriodSeconds: 120
in your FE spec to give the metadata store (BDBJE) enough time to close cleanly. Summary Checklist 1. Run 
SHOW FRONTENDS;
— Are there 3 Followers? 2. Check FE Pods — Do they have PVCs mounted to 
/opt/starrocks/fe/meta
? 3. Check Node Distribution — Use 
kubectl get pods -o wide
to confirm that no two FE pods are on the same node (confirming your Anti-Affinity is working). If you are using the StarRocks Operator, it is recommended to let the Operator manage the FE roles. Ensure you are using a recent version of the Operator, as older versions had issues with automatic FE re-joining after pod restarts. References • faq/operation_maintenance_faq.md
m
@Rocky quick clarification that changes the picture i tested FE and BE rolling restarts separately with a continuous row-count probe running during each: - Full FE rolling restart : completely clean Count kept climbing, max around 12s pause, no errors So FE quorum, FE persistence, and FE roles all look healthy, this isn't an FE/metadata quorum problem - Full BE rolling restart : around 47s ingestion stall with 
ERROR 1064: Backend node not found
. During the stall, one BE was down (being restarted) while the previously restarted BE still showed 
inBlacklist: true
, so 2 of 3 BEs were unusable at the same moment and Stream Load couldn't commit. So this looks like a BE rollout pacing / blacklist issue, not FE quorum. The rollout seems to take down the next BE before the FE has cleared the blacklist on the previous one. Specific questions: -What FE config controls the BE blacklist timeout : how long a BE stays 
inBlacklist
after recovery before the FE re-probes and re-adds it? Param name + default? -Is there a recommended way to pace a BE rolling restart so each BE is fully back and off the blacklist before the next is taken down? Does the operator's 
starRocksBeSpec
support anything like 
minReadySeconds
or a readiness gate? - Is 
graceful_exit_wait_for_frontend_heartbeat = true
on the BE enough on the shutdown side, or is there a recommended pattern to avoid the blacklist overlap during rollouts?
r
The behavior you’re seeing—where an ingestion stall occurs because one BE is restarting and another is stuck in the 
inBlacklist: true
state—is a known "pacing" issue during rolling updates. In a 3-replica setup, losing 2 BEs (one offline, one blacklisted) drops you below the required quorum (2 out of 3) for the 
PUBLISH
phase of ingestion. Here are the specific configurations to address this: 1. FE Config: BE Blacklist Timeout The FE parameter that controls how long a node remains on the automatic blacklist after a failure is 
black_host_history_sec
. * Parameter: 
black_host_history_sec
* Default: 60 seconds * Behavior: When an FE fails to connect to a BE (or a query times out), it blacklists that BE. The FE re-probes the BE based on this interval. If your rollout moves to the second BE before this 60s window expires for the first one, you effectively have 2 BEs out of rotation. * Recommendation: You can try reducing this to 
10
or 
20
seconds via 
ADMIN SET FRONTEND CONFIG ("black_host_history_sec" = "10");
to make the FE more aggressive in re-adding recovered BEs. 2. Operator Config: Rollout Pacing To prevent the operator from killing the next BE before the previous one is fully "off the blacklist," you should use 
minReadySeconds
. The StarRocks Operator supports 
minReadySeconds
in the 
starRocksBeSpec
. This field maps to the underlying Kubernetes 
StatefulSet
property. It forces Kubernetes to wait a specified number of seconds after the pod is "Ready" (passes readiness probes) before considering it "Available" and moving to the next pod. Example 
StarRocksCluster
YAML:
Copy code
yaml
spec:
  starRocksBeSpec:
    replicas: 3
    # Wait 90s after the BE is "Ready" before moving to the next BE pod.
    # This gives the FE enough time to clear the blacklist (60s default).
    minReadySeconds: 90 
    # ... other configs
3. Graceful Shutdown & Heartbeats You mentioned 
graceful_exit_wait_for_frontend_heartbeat = true
. This is highly recommended (available in v3.4.5+) because it ensures the BE doesn't actually exit until it receives a heartbeat from the FE acknowledging its "SHUTDOWN" status. * Why it helps: It prevents the FE from seeing a "sudden death" (which triggers the blacklist) and instead allows it to mark the node as 
EXITING
gracefully. * Is it enough? On its own, no. While it helps the start of the shutdown, it doesn't solve the recovery pacing. You still need 
minReadySeconds
to ensure the FE has fully "blessed" the new pod before the next one is killed. 4. Operator-wide "Safer Upgrades" If you are using a recent version of the StarRocks Operator (v1.10+), there is a global setting in the 
StarRocksCluster
spec called 
waitForFullRollout
.
Copy code
yaml
spec:
  # Ensures the operator verifies all pods in a component are at the 
  # current revision and healthy before proceeding with other logic.
  waitForFullRollout: true
Recommended Strategy 1. Set 
minReadySeconds: 90
 (or slightly higher than your 
black_host_history_sec
) in your 
starRocksBeSpec
. 2. Ensure 
graceful_exit_wait_for_frontend_heartbeat = true
 is in your 
be.conf
. 3. Monitor the blacklist: During your next rollout, run 
SHOW BACKEND BLACKLIST;
to see how long it takes for a node to disappear after its pod reaches the 
Running
state in Kubernetes. Adjust 
minReadySeconds
accordingly.
m
@Rocky I set 
minReadySeconds: 90
on the BE StatefulSet (confirmed 
.spec.minReadySeconds=90
), K8s server is v1.33 so it's supported — but during a BE rolling restart the pods still cycled only ~25s apart (be-2, be-1, be-0), not 90s, and I again saw 2 BEs in 
SHOW BACKEND BLACKLIST
at the same time. The BE spec uses 
podManagementPolicy: Parallel
. Is that why 
minReadySeconds
is being ignored during the rollout? And since 
podManagementPolicy
is immutable, what's the correct supported way to pace a BE rolling update in the operator so only one BE is ever down/blacklisted at a time?
r
You have correctly identified the root cause: 
podManagementPolicy: Parallel
prevents 
minReadySeconds
from being honored as a pacing mechanism during rolling updates. In Kubernetes, when a StatefulSet is set to 
Parallel
, it effectively tells the controller that pods do not have a strict order. While it speeds up initial deployment and scaling, it causes the rolling update controller to ignore the sequential "wait" logic that 
minReadySeconds
is intended to provide. To achieve a paced, sequential BE rollout where only one node is out of rotation at a time, follow these steps: 1. Change to 
OrderedReady
 You must explicitly set the 
podManagementPolicy
to 
OrderedReady
in your 
StarRocksCluster
(SRC) specification. This ensures Kubernetes waits for a pod to be "Ready" plus the 
minReadySeconds
duration before terminating the next pod.
Copy code
yaml
spec:
  starRocksBeSpec:
    replicas: 3
    # This is the crucial fix for sequential rollout pacing
    podManagementPolicy: OrderedReady 
    minReadySeconds: 90
2. Handle the Immutability Error The 
podManagementPolicy
field is immutable in Kubernetes StatefulSets. If you simply apply the YAML above, the StarRocks Operator will attempt to update the StatefulSet, but the Kubernetes API will reject it with a 
Forbidden
error. To apply this change, you must manually delete the BE StatefulSet without deleting your data: 1. Update your SRC CRD to include 
podManagementPolicy: OrderedReady
. 2. Delete the existing BE StatefulSet while keeping the pods and PVCs:
Copy code
bash
    # Replace 'starrocks-be' with your actual BE statefulset name
    kubectl delete statefulset <cluster-name>-be --cascade=orphan
Note: Using 
--cascade=orphan
ensures the BE pods keep running while you transition the controller. 3. The StarRocks Operator will see the StatefulSet is missing and recreate it with the new 
OrderedReady
policy. 4. Because the 
podManagementPolicy
is now 
OrderedReady
, subsequent rolling updates (triggered by image changes or config changes) will respect 
minReadySeconds
. 3. Verify the "Safe" Pacing Once 
OrderedReady
is active: * The rollout will proceed from the highest index to the lowest (e.g., 
be-2
-> 
be-1
-> 
be-0
). * Kubernetes will wait until 
be-2
is 
Ready
. * It will then wait an additional 90 seconds (your 
minReadySeconds
). * Only after that timer expires will it proceed to terminate 
be-1
. This 90-second buffer is enough time for the StarRocks FE to see that 
be-2
is back online, probe it, and remove it from the Blacklist (
black_host_history_sec
defaults to 60s) before the next node is taken offline. Summary Checklist * 
podManagementPolicy: OrderedReady
: Enables sequential logic. * 
minReadySeconds: 90
: Provides the buffer for FE blacklist clearance. * 
graceful_exit_wait_for_frontend_heartbeat: true
: (In 
be.conf
) Ensures the BE shuts down only after the FE acknowledges its exit. References • deployment/sr_operator.md • administration/management/BE_configuration.md • administration/management/graceful_exit.md References • administration/management/BE_configuration.md • administration/management/graceful_exit.md
4 Views
Open in Slack
PreviousNext
Powered by