@Rocky if I use helm chart and wanna upgrade to newer version should I first apply changes on frontend configs and after that separately to cn/be or I change all the tags at the same time and apply together and starrocks operator maintainces the order?

When using the StarRocks Operator (via Helm or direct YAML), the recommended practice is to apply the changes sequentially to ensure the cluster remains stable and available. Recommended Upgrade Order You should upgrade the components in the following sequence: 1. BE (Backend) and CN (Compute Node) first: These components are backward compatible with the Frontend (FE). 2. FE (Frontend) last: After all BEs and CNs are upgraded and confirmed to be "Alive." * FE Order: Within the FE group, upgrade the Follower nodes first, and upgrade the Leader node last. Why sequential is better than "all at once" While you could technically update all image tags in your

values.yaml

and run

helm upgrade

, doing so triggers a simultaneous reconciliation of the FE, BE, and CN resources. * Safety: If the FE starts its rolling restart before the BEs have finished their upgrade, you risk version incompatibility that can lead to service crashes. * Control: Applying changes separately allows you to verify that the BE nodes have successfully rejoined the cluster and are in an

Alive

state before you touch the critical metadata layer (FE). How to do it with Helm/Operator Instead of changing all tags at once in your

values.yaml

, follow these steps: 1. Update BE/CN first: Update only the

starRocksBeSpec.image

and

starRocksCnSpec.image

in your values or via

kubectl patch

.

bash
    kubectl -n starrocks patch starrockscluster <cluster-name> --type='merge' -p '{"spec":{"starRocksBeSpec":{"image":"starrocks/be-ubuntu:<new-version>"}}}'

2. Verify health: Check that the BE pods are running and use

SHOW BACKENDS;

in StarRocks to ensure they are

Alive

. 3. Update FE last: Update the

starRocksFeSpec.image

.

bash
    kubectl -n starrocks patch starrockscluster <cluster-name> --type='merge' -p '{"spec":{"starRocksFeSpec":{"image":"starrocks/fe-ubuntu:<new-version>"}}}'

Important Notes * Disable Tablet Balancing: For minor or major version upgrades, it is highly recommended to disable tablet clone/balancing before starting to prevent unnecessary data movement during the rolling restart.

sql
    ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
    ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "true");

* Consecutive Upgrades: Always follow the supported upgrade path (e.g., v2.2 -> v2.3 -> v2.4 -> v2.5). Do not skip minor versions unless explicitly unless the versions unless the release notes for a specific patch version allow it. References • deployment/upgrade.md • deployment/sr_operator.md

@Rocky but changing the tag on fespec starrocks operator by itself will start rolling update from followers?

Yes, the StarRocks Operator automates the rolling update process for the Frontend (FE) nodes, but it follows the standard Kubernetes StatefulSet update logic. Here is how the order works and what you should do to ensure a safe upgrade: 1. Component Order: Manual Control Recommended While the Operator maintains the cluster state, it does not automatically coordinate the sequence between different components (e.g., upgrading BEs before FEs) if you change all tags in a single

helm upgrade

. * Recommended Action: Change the image tags for BE/CN first, wait for them to be

Ready

and

Alive

, and only then update the FE image tag. This ensures backward compatibility is maintained throughout the process. 2. FE Internal Order: StatefulSet Logic When you update the

starRocksFeSpec.image

tag, the Operator updates the underlying FE StatefulSet. Kubernetes StatefulSets perform rolling updates in reverse ordinal order: * It will start with the highest index (e.g.,

fe-2

), wait for it to be

Ready

, then move to

fe-1

, and finally

fe-0

. 3. The "Followers First, Leader Last" Rule StarRocks' official recommendation is to upgrade Follower nodes first and the Leader node last to minimize metadata risks and election overhead. * How the Operator aligns: Since the Leader is typically

fe-0

(the first node created), the StatefulSet's reverse-order update (

fe-2

→

fe-1

→

fe-0

) naturally follows the "Follower first" rule in most cases. * If the Leader is not

fe-0

: If your current Leader is

fe-2

(e.g., due to a previous restart), the StatefulSet will update the Leader first. This will trigger a Leader election among the remaining older nodes. While this is generally handled by the system, it is less "graceful" than the recommended path. Best Practice for a Clean Upgrade To ensure the most stable upgrade using the Operator: 1. Check the current Leader: Run

SHOW FRONTENDS;

in your SQL client. 2. Align the Leader: If the Leader is not

fe-0

, you can manually restart the current Leader pod to force an election, or simply accept that a re-election will happen during the rollout. 3. Update BE/CN tags in your

values.yaml

and

helm upgrade

. 4. Verify BE health: Ensure all BEs are

Alive

. 5. Update FE tag in your

values.yaml

and

helm upgrade

. The Operator will then handle the rolling update of FEs from the highest index down to

fe-0

. Note: If you set the field

starRocksCluster.spec.starRocksFeSpec.waitForFullRollout: true

(in newer Operator versions), the Operator will ensure the FE update is 100% complete before it even attempts to touch other components, adding an extra layer of safety. References • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/Meta_recovery.md • deployment/deploy_manually.md • administration/management/graceful_exit.md

@Rocky if it will first update starrocks leader fe there can be risk of braking the system right?

To answer your concern about the risks of upgrading the Leader FE first: Does the Operator risk breaking the system? If you update the FE image tag, the StarRocks Operator uses a Kubernetes StatefulSet to perform a rolling update. This update happens in reverse ordinal order (e.g.,

fe-2

→

fe-1

→

fe-0

). While StarRocks documentation recommends "Followers first, Leader last," the process is generally safe even if the Leader is updated first because of how Leader election works: 1. Automatic Role Handoff: If the pod currently acting as the Leader (e.g.,

fe-2

) is restarted by the Operator for an upgrade, it first shuts down. 2. Safe Re-election: The remaining FE pods (

fe-1

and

fe-0

), which are still running the old version, will immediately elect a new Leader from among themselves. 3. Backward Compatibility: The upgraded pod (

fe-2

) will then start with the new version and join the cluster as a Follower. StarRocks is designed so that a newer-version Follower can sync metadata from an older-version Leader. 4. The "Leader Last" Goal: The goal is to ensure that by the time the cluster finally switches to a new-version Leader, all other Followers have already been upgraded. This prevents a situation where a new Leader writes metadata in a format that old-version Followers cannot understand. Is there a risk? The main risk occurs if you have a lack of quorum. If you only have two FEs and one is being upgraded, the other cannot become a Leader (it needs a majority). However, in a standard 3-node HA setup, the Operator's rolling update is safe. The "All-at-Once" Tag Change Conflict There is a specific behavior to be aware of when using the StarRocks Operator: * Manual Docs vs. Operator Logic: Standard StarRocks documentation (for manual installs) says to upgrade BEs first, then FEs. * Operator Logic: By default, the StarRocks Operator often prioritizes the FE upgrade or reconciles them in parallel. * Safety Tip: If you change all tags in your

values.yaml

at once, the Operator might start upgrading FEs and BEs simultaneously. To be safe, it is still best to: 1. Update the BE/CN image tags in your Helm

values.yaml

first and run

helm upgrade

. 2. Wait for all BE/CN pods to be

Ready

and

Alive

. 3. Then update the FE image tag and run

helm upgrade

again. Important Operator Setting:

waitForFullRollout

In recent versions of the StarRocks Operator, there is a parameter called

waitForFullRollout

(defaults to

false

). * If set to

true

, the Operator will ensure the FE StatefulSet is completely finished (all nodes upgraded and ready) before it starts updating any BE or CN nodes. * This is intended to prevent "cascading failures" where a broken FE version might otherwise trigger a BE upgrade that could take the whole cluster down. Summary Recommendation: To avoid any risk, apply changes to BE/CN first and FE last in two separate

helm upgrade

commands. This follows the official StarRocks upgrade path and ensures that the Leader election and metadata synchronization remain stable throughout the process. References • deployment/upgrade.md • faq/operation_maintenance_faq.md • administration/Meta_recovery.md • deployment/deploy_manually.md • administration/management/graceful_exit.md References • faq/operation_maintenance_faq.md • deployment/upgrade.md • administration/management/graceful_exit.md

@Rocky and what about rollback still cn/be first and fe last again?

For a rollback (downgrade), the order is the exact reverse of the upgrade process. You must downgrade the components in the following sequence: 1. Recommended Rollback Order 1. FE (Frontend) first: Upgrade order was BE then FE; rollback order is FE then BE. 2. BE/CN (Backend/Compute Node) last: Only after the FEs are successfully rolled back and healthy. Why FE first? Downgrading in the wrong order (e.g., rolling back BEs while the FE is still on the newer version) can lead to protocol or metadata incompatibilities that may cause the service to crash. An older-version FE is generally designed to be safe while communicating with newer-version BEs during a transition, but the reverse is not always true for rollbacks. 2. How to handle this with Helm/Operator Even if you are using

helm rollback

or updating the tags in your

values.yaml

, you should still apply the changes sequentially to mimic the manual safety steps: 1. Roll back FE tags first: Update only the

starRocksFeSpec.image

to the previous version in your Helm values and run

helm upgrade

. 2. Verify FE health: Ensure all FE pods are

Ready

and a Leader is elected. 3. Roll back BE/CN tags last: Once FEs are stable, update the

starRocksBeSpec.image

and

starRocksCnSpec.image

tags and run

helm upgrade

again. 3. Critical FE Internal Order The Operator will roll back FE pods in reverse ordinal order (e.g.,

fe-2

then

fe-1

then

fe-0

). * This naturally follows the rule to downgrade Follower FEs first and the Leader FE last. * If your Leader is

fe-0

, the rolling update will handle the followers first. If the Leader is not

fe-0

, it is safer to restart the Leader pod manually before starting the rollback to force the Leader role to

fe-0

. 4. Pre-Rollback Checklist * Metadata Snapshot: Before rolling back FEs, it is highly recommended to create a metadata snapshot (image) to ensure you have a clean recovery point.

sql
    ALTER SYSTEM CREATE IMAGE;

* Disable Balancing: Just like the upgrade, disable tablet balancing before starting the rollback to prevent data movement during pod restarts.

sql
    ADMIN SET FRONTEND CONFIG ("disable_balance" = "true");
    ADMIN SET FRONTEND CONFIG ("disable_colocate_balance" = "true");

* Version-Specific Configs: If you are rolling back across minor versions (e.g., v3.x to v2.5), check the documentation for specific compatibility flags like

ignore_unknown_log_id

. References • deployment/downgrade.md • deployment/upgrade.md • administration/Meta_recovery.md References • deployment/downgrade.md • deployment/upgrade.md