http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> when I do rolling update of a versi...
# questions-and-troubleshooting
z
@Rocky when I do rolling update of a version (patch) it will fail ongoing queries?
✅ 1
r
Whether a rolling update (patch) will fail ongoing queries depends on your StarRocks version and how you stop the nodes. 1. Using Graceful Exit (Recommended) Starting from v3.3, StarRocks introduced a Graceful Exit mechanism specifically to minimize the impact on running queries and load jobs during rolling updates. * How to use it: Instead of a standard stop, use the 
-g
flag: * 
stop_fe.sh -g
* 
stop_be.sh -g
* What happens: * The node immediately stops accepting new queries or fragments. * It waits for ongoing queries to complete within a controlled time window. * FE Behavior: Follower FEs wait for active sessions to finish (default timeout 60 seconds). * BE Behavior: BEs wait for existing query fragments to finish (default timeout is 20 seconds, controlled by 
loop_count_wait_fragments_finish
). 2. Impact Based on Query Duration Even with Graceful Exit, the impact depends on how long your queries run: * Short queries (< 20s): Typically complete normally without failure. * Medium queries (20s - 60s): Should complete if you have tuned the wait timeouts, otherwise they may be cancelled and require a retry. * Long queries (> 60s): These are likely to be terminated if they exceed the maximum graceful wait window. 3. Version Considerations * v3.4 and later: This is the most stable version for rolling updates. The FE is "aware" that a BE is in an "exiting" state and won't prematurely mark it as dead, allowing for a smoother transition. * v3.3: Graceful exit exists but has some limitations where the FE might still mark a BE as dead prematurely, potentially failing some queries. * Before v3.3: StarRocks does not have a formal graceful exit. Stopping a node will likely fail any queries currently being processed by that specific node. Summary of Best Practices 1. Upgrade to v3.4+ for the best rolling update experience. 2. Use 
stop_be.sh -g
and 
stop_fe.sh -g
 during your update process. 3. Check for long-running queries before starting the update, as these are the most likely to fail if they exceed the graceful timeout. 4. Use a Load Balancer (like F5 or HAProxy) in front of your FEs so that when an FE enters graceful exit (returning HTTP 500 on its health check), the balancer stops sending new traffic to it immediately. References • administration/management/graceful_exit.md • administration/management/BE_configuration.md • faq/Sql_faq.md • using_starrocks/caching/query_cache.md
z
@Rocky I do it via helm in k8s cluster
r
Yes, when performing a rolling update via Helm in a Kubernetes cluster, ongoing queries can be impacted, but the risk is significantly reduced if you are on a recent version of StarRocks (v3.4+). The StarRocks Kubernetes Operator and Helm chart are designed to handle this gracefully using built-in preStop hooks and the Graceful Exit mechanism. How it Works in Kubernetes When you trigger a rolling update (e.g., by changing the image tag in your Helm values), Kubernetes follows this sequence for each pod: 1. Pre-stop Hook Triggers: The StarRocks Operator automatically adds pre-stop scripts to your pods: * FE: 
/opt/starrocks/fe_prestop.sh
(sends 
SIGUSR1
to trigger graceful exit). * BE/CN: 
/opt/starrocks/be_prestop.sh
(sends 
SIGTERM
to trigger graceful exit). 2. Graceful Exit Phase: * The node stops accepting new query fragments. * It waits for ongoing fragments to finish. * FE Awareness (v3.4+): The Frontend is informed that the Backend is "Exiting." It will wait for the BE to finish its work rather than immediately marking it as 
DEAD
and failing the query. 3. Termination Grace Period: By default, the operator sets 
terminationGracePeriodSeconds
to 120 seconds. If the queries don't finish within this window, Kubernetes will forcibly kill the pod (
SIGKILL
), and those queries will fail. Will my queries fail? It depends on the StarRocks Version and Query Duration: StarRocks Version | Impact on Queries v3.4 and later | Best Experience. Supports extended wait windows. FE knows the BE is exiting and won't kill the query prematurely. Most queries under 60-120s will succeed. v3.3 | Basic Support. Has some "flaws" where the FE might still mark a BE as 
DEAD
after ~15s of heartbeat failure, potentially failing mid-length queries. Before v3.3 | High Risk. Rolling updates will likely fail any query currently being processed by the pod being restarted. Recommendations for Helm Users To ensure the smoothest rolling update in K8S: 1. Check Version: Ensure you are running v3.4.0 or later. 2. Adjust Grace Period: If you have very long-running queries (e.g., 5-minute ETLs), increase the 
terminationGracePeriodSeconds
in your 
StarRocksCluster
CR or Helm values to be longer than your longest query.
Copy code
yaml
    # Example snippet for Helm values.yaml
    starrocksCluster:
      beSpec:
        terminationGracePeriodSeconds: 300 # 5 minutes
3. BE Configuration: For v3.4.5+, ensure 
graceful_exit_wait_for_frontend_heartbeat
is set to 
true
(it allows the BE to confirm the FE has seen its exit status before shutting down). 4. Use a Service/Load Balancer: Ensure your application connects to the FE via a Kubernetes Service. During a rolling update, the FE pre-stop hook makes the FE return HTTP 500 on its health check, causing the K8S Service to stop routing new traffic to it ~15 seconds before it actually shuts down. References • administration/management/graceful_exit.md • starrocks-kubernetes-operator/doc/api.md • administration/management/BE_configuration.md • quick_start/helm.md References • administration/management/graceful_exit.md • quick_start/helm.md
z
@Rocky whats starrocksCluster: componentValues: "3.5.15"?
r
The field 
starrocksCluster: componentValues
in your Helm 
values.yaml
is a global configuration block used to apply settings to all StarRocks components (FE, BE, and CN) simultaneously. When you see 
"3.5.15"
associated with it, it refers to the StarRocks version (image tag) you are deploying. 1. What does it do? Instead of manually defining the same settings (like image repository, tag, or resource limits) inside individual 
starrocksFESpec
and 
starrocksBeSpec
blocks, you can define them once under 
componentValues
. This ensures your entire cluster stays on the same version and shares the same base configuration. 2. Correct Syntax In most versions of the StarRocks Helm chart, 
componentValues
is an object, not a single string. If you want to set the version to 
3.5.15
globally, your values file should look like this:
Copy code
yaml
starrocks:
  starrocksCluster:
    componentValues:
      image:
        tag: "3.5.15"
3. Precedence Rules * Centralized Control: Settings in 
componentValues
apply to FE, BE, and CN. * Specific Overrides: If you define a setting inside a specific component spec (e.g., 
starrocksFESpec.image.tag
), it will overwrite the global value in 
componentValues
for that specific component. * Non-Merging: Note that if you define a setting in the component spec, it usually replaces the corresponding section in 
componentValues
rather than merging with it. 4. About Version 3.5.15 StarRocks v3.5.15 is a stable patch release in the 3.5 LTS (Long Term Support) branch. In the context of your previous question about rolling updates: * Using 
3.5.15
via this field ensures that all nodes are updated to the same patch version during the Helm upgrade. * Because it is a version higher than v3.4, it fully supports the Graceful Exit and FE-Awareness features discussed earlier, making your rolling update via Helm very safe for ongoing queries. References • quick_start/helm.md • administration/management/graceful_exit.md • administration/management/BE_configuration.md • faq/Sql_faq.md • using_starrocks/caching/query_cache.md References • quick_start/helm.md
z
@Rocky correct approach is to upgrade helm value of cn deploy it and after that upgrade version of fe to minimize issues while rolling updates and in that case I should set component values 3.5.16 and cn 3.5.16 and after everything will work set fe ?
r
Yes, you have the right general idea, but with Helm and the StarRocks Operator, there are two ways to handle this. 1. The Upgrade Order You are correct: Upgrade CNs and BEs first, then FEs. * Why: BEs and CNs are designed to be backward compatible with older FEs. If you upgrade the FE first, an older BE might not understand new metadata or instructions sent by the new FE, leading to query failures or crashes. * FE Order: Within the FE group, you should upgrade all Follower FEs first, and finally the Leader FE. 2. How to do this with Helm If you want to manually control the sequence to be extra safe, do not use 
componentValues
for the version in the first step. 
componentValues
acts as a global default—if you set it to 
3.5.16
, the Operator will see that all components need an update and will start the process for everyone. Step-by-Step Approach (Manual Sequence) Step 1: Upgrade CN and BE only In your 
values.yaml
, keep the FE version at 
3.5.15
and explicitly set the CN/BE to 
3.5.16
.
Copy code
yaml
starrocks:
  starrocksBeSpec:
    image:
      tag: "3.5.16"
  starrocksCnSpec:
    image:
      tag: "3.5.16"
  starrocksFESpec:
    image:
      tag: "3.5.15" # Keep old version
Run 
helm upgrade
. Wait for all BE/CN pods to reach the 
Running
state and for the Operator to show the cluster as 
running
. Step 2: Upgrade FE Once BEs are stable, update the FE tag in your `values.yaml`:
Copy code
yaml
starrocks:
  starrocksFESpec:
    image:
      tag: "3.5.16"
Run 
helm upgrade
again. 3. Alternative: Let the Operator handle it The StarRocks Operator is "smart." If you update all versions to 
3.5.16
at once in a single `helm upgrade`: 1. The Operator detects the changes. 2. It is programmed to follow the correct lifecycle (it will typically start with BEs/CNs). 3. However, for production environments where you want zero-downtime and maximum control over query impact, the Step-by-Step manual approach above is preferred. Important Configuration Note If you are using 
componentValues
, it is usually used like this to avoid repetition:
Copy code
yaml
starrocks:
  starrocksCluster:
    componentValues:
      image:
        repository: starrocks/be-ubuntu # Example
If you put 
tag: "3.5.16"
here, it will apply to everything immediately. To stagger the upgrade, define the tags inside 
starrocksBeSpec
, 
starrocksCnSpec
, and 
starrocksFESpec
individually as shown in the Step-by-Step guide. References • deployment/upgrade.md • deployment/sr_operator.md • quick_start/helm.md • administration/management/BE_configuration.md • faq/Deploy_faq.md References • deployment/sr_operator.md • deployment/upgrade.md
z
@Rocky does not helm automatically decides what pods to upgrade first?
r
No, Helm does not automatically decide the upgrade order between different component types (like BE vs. FE). When you run 
helm upgrade
and change the version tags for both FE and BE at the same time, Helm simply updates the 
StarRocksCluster
resource in Kubernetes. The StarRocks Operator then sees that both the FE and BE "desired states" have changed and will likely start updating both groups at the same time. Why you should do it manually (2-Step Process) Even though the Operator manages the rolling update of pods within a group (e.g., updating BE-0, then BE-1, then BE-2), it does not strictly enforce that the BE group must finish before the FE group starts unless you stagger the configuration changes. To minimize issues, follow this "staggered" Helm approach: Step 1: Upgrade BEs and CNs first Update only the BE and CN tags in your 
values.yaml
. * Why: BEs are backward compatible. A new BE can talk to an old FE Leader without issues. * Action:
Copy code
yaml
    starrocksBeSpec:
      image: { tag: "3.5.16" }
    starrocksFESpec:
      image: { tag: "3.5.15" } # Keep old version for now
* Command: 
helm upgrade <release-name> starrocks/kube-starrocks -f values.yaml
* Check: Wait until 
kubectl get pods
shows all BEs/CNs are 
Running
and healthy. Step 2: Upgrade FEs Once the backends are stable on the new version, update the FE tag. * Why: FEs are the "brain." If you upgrade them first, they might send new commands or metadata formats that the older BEs don't understand, causing query failures. * Action: Change 
starrocksFESpec
tag to 
"3.5.16"
. * Command: 
helm upgrade <release-name> starrocks/kube-starrocks -f values.yaml
Summary of Component Order 1. CN/BE Nodes: Any order. 2. Follower FE Nodes: One by one. 3. Leader FE Node: Always last. (The Operator usually handles the internal FE order correctly, but it cannot know when you want to "start" the FE phase vs. the BE phase unless you separate the Helm calls). Is there a "Global" way? If you set the version in 
starrocksCluster.componentValues
, it applies to all components immediately. If you use this field, the Operator will likely trigger updates for FE, BE, and CN simultaneously, which increases the risk of query interruption during the period where the versions are mismatched. For production, the 2-step manual tag update is the safest approach. References • deployment/upgrade.md • deployment/sr_operator.md • administration/management/graceful_exit.md References • deployment/upgrade.md • deployment/sr_operator.md • administration/management/graceful_exit.md • quick_start/helm.md • administration/management/BE_configuration.md • faq/Deploy_faq.md • deployment/upgrade.md • deployment/sr_operator.md • administration/management/graceful_exit.md • quick_start/helm.md • administration/management/BE_configuration.md • faq/Deploy_faq.md References • deployment/upgrade.md • deployment/sr_operator.md
z
@Rocky why fe followers first and fe leader last?
r
Upgrading the FE Followers first and the FE Leader last is a critical safety requirement for StarRocks due to how metadata and cluster leadership work. 1. Metadata Forward-Compatibility StarRocks uses BDBJE (Berkeley DB Java Edition) to manage its metadata through an "Edit Log." * The Rule: A newer version of StarRocks can almost always read and replay metadata written by an older version. However, an older version cannot read metadata written by a newer version. * The Risk: If you upgrade the Leader first, it may start writing metadata in a new format or with new attributes specific to the new version. The remaining Followers (still on the old version) will try to synchronize this metadata, fail to parse it, and potentially crash or become disconnected. 2. Leadership Stability and Elections When you stop an FE node for an upgrade, a re-election may occur if that node was the Leader. * Ready Successors: By upgrading the Followers first, you ensure that when it eventually comes time to upgrade the Leader, the nodes standing by to take over (the Followers) are already running the new version. * Smooth Transition: If you upgrade the Leader last, it will step down, and one of the already-upgraded Followers will take over as the new Leader. Since this new Leader is already on the new version, the transition is seamless. 3. "Helper" Availability During an upgrade, a node might need to synchronize its entire metadata state (a "checkpoint") from an existing node. * The Leader acts as the primary source of truth. By keeping the Leader on a stable, older version while Followers cycle through their upgrades, you ensure there is always a consistent source of metadata for the Followers to sync from as they come back online. Summary of the Safe Workflow: 1. Upgrade BEs/CNs: They are backward compatible with the FE. 2. Upgrade FE Followers (one by one): They will pull the "old" metadata from the current Leader and run it in a compatibility mode. 3. Upgrade FE Leader: It will stop, a new election will occur (choosing one of the already-upgraded Followers), and the cluster will complete its transition to the new version. References • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md • faq/operation_maintenance_faq.md References • deployment/upgrade.md • faq/operation_maintenance_faq.md • deployment/deploy_manually.md • administration/Meta_recovery.md
z
@Rocky ok and how to easily make fe0 as a leader when fe2 is the leader and wanna do safe rolling update
r
In StarRocks, there is no SQL command to "assign" leadership to a specific node. Leadership is determined by an automated election among all Follower FEs. If you are using the StarRocks Operator on Kubernetes, it is designed to be aware of the Leader and should automatically attempt to upgrade it last. However, if you want to manually ensure fe-0 is the Leader before starting your Helm upgrade (to align with the standard Kubernetes StatefulSet rolling update order of 
fe-2
-> 
fe-1
-> 
fe-0
), you can use the "Restart" method. How to move Leadership to fe-0 1. Check the current Leader: Run this in your SQL client:
Copy code
sql
    SHOW FRONTENDS;
Look at the 
Role
column to see which node is 
LEADER
. 2. Force a Re-election: If 
fe-2
or 
fe-1
is the leader, delete that pod. Kubernetes will immediately restart it, but in the few seconds it is down, the remaining FEs will hold an election.
Copy code
bash
    # If fe-2 is the leader:
    kubectl delete pod <fe-2-pod-name> -n <namespace>
3. Verify and Repeat: Wait for the deleted pod to come back and show as 
Alive: true
in 
SHOW FRONTENDS;
. If the leader is still not 
fe-0
, repeat the process for the current leader until 
fe-0
wins the election. Why this matters for your Upgrade By making fe-0 the leader, you align the "Leader Last" rule with the default behavior of Kubernetes StatefulSets: * Default K8s Order: Kubernetes typically updates pods in reverse order: 
fe-2
, then 
fe-1
, then 
fe-0
. * The Result: Your Followers (
fe-2
and 
fe-1
) will be upgraded first while the Leader (
fe-0
) remains on the old version. Only when the followers are healthy and running the new version will Kubernetes finally reach 
fe-0
. Summary of the "Safe" Rolling Upgrade 1. Staggered Tags: Use the 2-step Helm approach (CN/BE first, then FE) as discussed. 2. Leadership check: Ensure the node that Kubernetes will update last (usually 
fe-0
) is the current Leader before you update the FE image tag in Helm. 3. Monitor: After running 
helm upgrade
, watch the pods. If a follower fails to start, do not proceed to the next node. References • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md • administration/management/graceful_exit.md References • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md • administration/management/graceful_exit.md • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md • administration/management/graceful_exit.md • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md
z
@Rocky yes I use starrocks operator
r
When using the StarRocks Operator, the FE nodes are managed as a Kubernetes StatefulSet. StatefulSets have a fixed update order: they always update pods in reverse ordinal order (e.g., 
fe-2
, then 
fe-1
, and finally 
fe-0
). Because StarRocks requires the Leader to be upgraded last, you should ensure that 
fe-0
is the Leader before you start the upgrade. If 
fe-2
or 
fe-1
is the Leader when you trigger the upgrade, Kubernetes will kill and upgrade that Leader pod first, which could cause metadata synchronization issues with the older followers. How to ensure a safe Operator upgrade 1. Check current leadership Run the following SQL command in your StarRocks cluster:
Copy code
sql
SHOW FRONTENDS;
Check the 
Role
column. If 
fe-0
is the 
LEADER
, you are ready to upgrade. 2. Move the Leader to 
fe-0
(if necessary) If another node (like 
fe-2
) is the Leader, you can force it to step down: 1. Delete the current Leader pod:
Copy code
bash
   kubectl delete pod <name-of-leader-pod> -n <namespace>
2. Wait for a new election: The remaining FE pods will elect a new leader. 3. Repeat: Check 
SHOW FRONTENDS;
again. If 
fe-0
isn't the leader yet, delete the pod that is currently the leader until 
fe-0
wins the election. 3. Trigger the Helm upgrade Once 
fe-0
is the Leader, update your 
values.yaml
and run:
Copy code
bash
helm upgrade <release-name> starrocks/kube-starrocks -f values.yaml
Kubernetes will now upgrade 
fe-2
(Follower), then 
fe-1
(Follower), and finally 
fe-0
(Leader), which perfectly follows the StarRocks safety requirement. Does the Operator handle this automatically? The Operator manages the rolling update process, but it relies on the underlying Kubernetes StatefulSet controller for the pod restart sequence. It does not currently "re-index" the pods or manually swap leadership roles to match the update order. Therefore, ensuring 
fe-0
is the Leader is the recommended manual "pre-flight" check for a production upgrade. References • deployment/upgrade.md • deployment/sr_operator.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md References • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md
z
@Rocky what about running cn first and fe last via starrocks operator it cant handle that order as well and first cn helm value should be updated and after that fe manually?
r
To ensure a safe upgrade using the StarRocks Operator, you should follow a two-step process. The Operator manages the components (FE, BE, CN) as separate controllers, and the safest way to enforce the required order is to update the images in stages. The Recommended Order As per the official documentation, BEs and CNs are backward compatible with FEs. Therefore, the sequence must be: 1. Upgrade CNs and BEs first. 2. Upgrade FEs last. How to do this with Helm and the Operator If you update all image tags (FE, BE, and CN) in your 
values.yaml
at once and run 
helm upgrade
, the Operator will attempt to update all components. While the Operator tries to be safe, it does not strictly guarantee that it will finish all CNs before starting the first FE unless you stagger the updates. Step 1: Upgrade CN/BE only Update your 
values.yaml
with only the new CN and BE image tags. Leave the FE image tag on the old version.
Copy code
yaml
starrocksBeSpec:
  image: starrocks/be-ubuntu:3.x.x # New Version
starrocksCnSpec:
  image: starrocks/be-ubuntu:3.x.x # New Version
starrocksFeSpec:
  image: starrocks/fe-ubuntu:2.x.x # KEEP OLD VERSION
Run 
helm upgrade
. Wait for all CN and BE pods to reach the 
Running
and 
Ready
state. Step 2: Upgrade FE (with Leader check) Once the CNs and BEs are healthy: 1. Check Leadership: Run 
SHOW FRONTENDS;
and ensure 
fe-0
 is the Leader (as discussed previously, since Kubernetes upgrades 
fe-2
-> 
fe-1
-> 
fe-0
). 2. Update FE tag: Now update the FE image tag in your 
values.yaml
.
Copy code
yaml
    starrocksFeSpec:
      image: starrocks/fe-ubuntu:3.x.x # New Version
3. Apply: Run 
helm upgrade
again. Why manually stagger? * Heartbeat Compatibility: A new BE/CN can always talk to an old FE. However, an old BE/CN might not correctly handle new protocol features or metadata formats sent by a newer FE. * Operator Logic: By default, the StarRocks Operator treats each component's specification independently. While there is an internal flag in some versions to make BEs wait for FEs to be "Ready" (
waitFeReady
), this is primarily for fresh installs to ensure BEs have a leader to register with. It does not automatically enforce the "CN first, FE last" rule for version upgrades. Summary Yes, you should update the CN/BE values first, apply the change, verify they are healthy, and only then update the FE values. This is the standard "Best Practice" for production StarRocks clusters on Kubernetes. References • deployment/upgrade.md • deployment/sr_operator.md • administration/management/graceful_exit.md • faq/operation_maintenance_faq.md References • deployment/upgrade.md • deployment/sr_operator.md • deployment/downgrade.md
2 Views
Open in Slack
PreviousNext
Powered by