http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> what am I looking forward to `Verif...
# questions-and-troubleshooting
g
@Rocky what am I looking forward to 
Verify that the image file has synchronized to all Follower FEs by checking the fe.log on the Leader FE.
r
To verify that the metadata image file has synchronized from the Leader FE to the Follower FEs, you should look for specific log patterns and then verify the state via SQL. 1. What to look for in the Leader 
fe.log
 When you run 
ALTER SYSTEM CREATE IMAGE
, the Leader FE creates a new snapshot. Followers then pull this file via HTTP. On the Leader FE, you are looking for: * Phase 1: Image Creation (Leader-only)
Copy code
text
    INFO (...) [Checkpoint.runAfterCatalogReady():xxx] triggering a new checkpoint manually...
    ...
    INFO (...) [Checkpoint.saveImage():xxx] finished save image /path/to/fe/meta/image/image.xxxxxx.
This confirms the Leader has successfully written the new image to its own disk. * Phase 2: Follower Synchronization StarRocks Followers pull the image from the Leader over the FE's HTTP port (default 
8030
). In the Leader's 
fe.log
(or 
fe.out
depending on your logging setup), you may see HTTP access logs or INFO logs indicating a transfer:
Copy code
text
    INFO (...) [ImageServlet.doGet():xxx] get image: image.xxxxxx from /<follower-ip>
*Note: If your logging level is set to INFO, you might not see a verbose "sync complete" message for every follower. The most reliable way to confirm they have actually loaded it is the next step.* 2. The Final Verification (SQL) After the logs indicate the image was created, run this command from your MySQL client:
Copy code
sql
SHOW PROC '/frontends'\G
Look for the 
ReplayedJournalId
column. * If the Followers' 
ReplayedJournalId
is nearly identical to the Leader's, they are in sync. * If a Follower is lagging significantly or has a 
Join: false
status, it may still be downloading or failing to load the new image. ────────── Operator Compatibility (v1.11.3 to v1.11.4) Regarding your upgrade from StarRocks 4.0.2 to 4.0.9: 1. Is 1.11.4 required? It is not strictly required for 4.0.9 functionality, but it is highly recommended. 2. The "Graceful Shutdown" Fix: Operator v1.11.4 contains critical fixes for the 
preStop
hook (graceful shutdown). In previous versions, the 
-g
(graceful) flag was sometimes added incorrectly for certain versions, which could lead to data loss or long recovery times if a Pod restarted during a heavy load. 3. Important CRD Note: If you haven't done so since moving to the 1.11.x branch, remember that Operator 1.11.2+ introduced a new field (
podManagementPolicy
) in the CRD. * If you upgrade the Helm chart from 1.11.3 to 1.11.4, you must first manually apply the updated CRD:
Copy code
bash
        kubectl apply -f <https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml>
* Failure to do this can cause the Helm upgrade to fail because the new chart expects a CRD schema that your Kubernetes cluster doesn't recognize yet. Recommendation: Upgrade the Operator to 1.11.4 first, ensure the operator pods are stable, and then proceed with the StarRocks 4.0.9 image upgrade. References • release_notes/release-3.5.md • release_notes/release-2.5.md • release_notes/release-3.3.md
g
@Rocky looking at the fe.log on the leader, I don't see anything with 
checkpoint
how can I verify the 
ALTER SYSTEM CREATE IMAGE
did anything?
r
To verify the image synchronization and handle your upgrade path, here is the breakdown of what you should look for in the logs and the recommended upgrade strategy. 1. Verifying the Metadata Image If you don't see the word 
checkpoint
in your 
fe.log
, it is likely because you are searching for the wrong keyword or the asynchronous process hasn't finished yet. When you run 
ALTER SYSTEM CREATE IMAGE
, you should search the Leader FE's 
fe.log
 for the following patterns: * To check if it started: Search for 
image
or 
triggering
. * 
INFO (...) [Checkpoint.runAfterCatalogReady():...] triggering a new checkpoint manually...
* 
INFO (...) [Checkpoint.saveImage():...] begin to generate new image: image.xxxxxx
* To check if it finished locally: * 
INFO (...) [Checkpoint.saveImage():...] finished save image /path/to/fe/meta/image/image.xxxxxx
* To verify synchronization to Followers: This is the specific log you are "looking forward to" according to the documentation: * 
INFO (...) [ImageServlet.doGet():...] get image: image.xxxxxx from /<follower-ip>
* OR: 
push image.xxxxxx from subdir [] to other nodes. totally X nodes, push successful X nodes
Alternative Verification (SQL): If logs are rotated or hard to parse, run this command:
Copy code
sql
SHOW PROC '/frontends'\G
Check the 
ReplayedJournalId
. All Follower FEs should have a 
ReplayedJournalId
that is very close to (or identical to) the Leader's. If a follower is lagging by thousands of IDs, the image might still be transferring. ────────── 2. Operator Upgrade (v1.11.3 to v1.11.4) While not strictly required by the StarRocks binary version (4.0.9), it is highly recommended to upgrade to Operator v1.11.4 before upgrading your StarRocks cluster. Why? * Graceful Shutdown Fix: Operator v1.11.4 contains a critical fix for the 
preStop
hook (graceful shutdown). In previous versions (including 1.11.3), there was a bug where the 
-g
(graceful) flag was handled incorrectly for certain versions. * Data Safety: During a StarRocks upgrade (4.0.2 to 4.0.9), the Operator will restart your Pods one by one. Without the fix in 1.11.4, a Pod might not shut down gracefully, which could lead to longer recovery times or potential data loss if a heavy data import is running during the upgrade. Upgrade Order Recommendation: 1. Upgrade Operator: Upgrade your Helm chart or YAML to v1.11.4. _ _Note:* If you are moving from a version earlier than 1.11.2, ensure you manually apply the updated CRD first (
kubectl apply -f ...starrocks.com_starrocksclusters.yaml
) as a new field 
podManagementPolicy
was introduced. 2. Run Create Image: Execute 
ALTER SYSTEM CREATE IMAGE
on the Leader FE and verify it has synced (as described above). 3. Upgrade StarRocks: Change the image tag in your 
StarRocksCluster
spec from 
4.0.2
to 
4.0.9
. Summary * Logs: Search for 
push image
or 
finished save image
in 
fe.log
. * Operator: Go to 1.11.4 first. It ensures that when your BE/FE pods restart for the 4.0.9 upgrade, they do so safely. References • deployment/deploy_manually.md • administration/Meta_recovery.md
g
@Rocky ok i have the operator image available, after I change the reference in the helm yaml, how do i get it to start using the new one?
r
To apply the new operator image (v1.11.4) and start using it, follow these three steps. Step 1: Update the CRD (Crucial) Before running the Helm upgrade, you must manually update the Custom Resource Definitions (CRDs). The Operator 1.11.x branch introduced new fields (like 
podManagementPolicy
) that the older 1.11.3 CRD might not have. Run this command:
Copy code
bash
kubectl apply -f <https://raw.githubusercontent.com/StarRocks/starrocks-kubernetes-operator/main/deploy/starrocks.com_starrocksclusters.yaml>
Step 2: Run the Helm Upgrade Once the CRD is updated, use the 
helm upgrade
command to point to your updated 
values.yaml
(where you've changed the image tag to 
1.11.4
). Replace 
<release-name>
and 
<namespace>
with your actual deployment values:
Copy code
bash
# If you are using the 'kube-starrocks' chart:
helm upgrade <release-name> starrocks/kube-starrocks -f values.yaml -n <namespace>

# OR if you are just upgrading the standalone operator chart:
helm upgrade <release-name> starrocks/operator -f values.yaml -n <namespace>
Step 3: Verify the Operator Pod The upgrade will trigger a rolling restart of the operator pod. You can verify it is using the new image by checking the pod status: 1. Watch the restart:
Copy code
bash
    kubectl get pods -n <namespace> | grep operator
2. Confirm the image version:
Copy code
bash
    kubectl describe pod <operator-pod-name> -n <namespace> | grep Image:
You should see the tag 
1.11.4
in the output. What happens next? Once the Operator is running version 1.11.4, it will "take over" the management of your StarRocks cluster. * The Operator upgrade itself will NOT restart your StarRocks FE/BE pods. * The StarRocks pods will only restart later when you change the StarRocks image tag (to 4.0.9) in your 
StarRocksCluster
YAML. Because you are now on Operator 1.11.4, those restarts will use the fixed, safer graceful shutdown logic. References • deployment/sr_operator.md • deployment/helm.md • quick_start/helm.md
g
@Rocky I suppose if i only change the reference to the operator in my values.yaml then run 
helm upgrade <name> starrocks/kube-starrocks -f values.yaml -n <namespace>
it'll just do the operator. Then I can do the same with the CNs, then last with the FEs. Is that a good plan?
r
Yes, your plan is excellent and follows the recommended safety protocols for a StarRocks upgrade on Kubernetes. Upgrading in discrete steps via Helm allows you to monitor the health of the cluster at each stage rather than changing everything at once. Here is the recommended execution order for your plan: 1. Upgrade the Operator (Infrastructure) * Action: Change the operator image tag to 
1.11.4
in 
values.yaml
and run 
helm upgrade
. * Why: As discussed, this ensures the management layer is ready with the latest "Graceful Shutdown" logic and CRD support before the actual StarRocks pods are touched. * Verification: Ensure the operator pod is 
Running
and the logs show no errors. 2. The "Pre-Upgrade" Safety Step (Metadata) * Action: 1. Run 
ALTER SYSTEM CREATE IMAGE;
. 2. Verify it has synced to followers (check 
fe.log
or 
SHOW PROC '/frontends'
). * Why: This ensures that if the FE upgrade encounters an issue, your metadata is already checkpointed and synchronized across all nodes. 3. Upgrade the CNs/BEs (Data/Compute Layer) * Action: Change the CN image tag to 
4.0.9
in 
values.yaml
and run 
helm upgrade
. * Why: StarRocks requires that BEs/CNs are upgraded before FEs. This is because newer BE/CN versions are backward-compatible with older FE versions, but the reverse is not always true. * Operator Behavior: The Operator will perform a rolling restart of your CN pods one by one. It will wait for each pod to become 
Ready
before moving to the next. 4. Upgrade the FEs (Control/Metadata Layer) * Action: Finally, change the FE image tag to 
4.0.9
and run 
helm upgrade
. * Operator Behavior: The Operator is "FE-aware." It will automatically upgrade the Follower FEs first and then perform a switchover/restart for the Leader FE last. This minimizes downtime for the cluster's control plane. Summary of your "Good Plan" 1. Helm Update (Operator): 
helm upgrade ...
(only operator tag changed). 2. SQL: Trigger and verify Image creation. 3. Helm Update (CNs): 
helm upgrade ...
(only CN tag changed). 4. Helm Update (FEs): 
helm upgrade ...
(only FE tag changed). This "step-by-step" Helm approach is much safer than changing all tags at once, as it allows you to pause if you notice any performance regressions after the CN upgrade but before you touch the critical FE metadata layer that manages your metadata (FE) your metadata. References • deployment/upgrade.md • deployment/sr_operator.md • deployment/helm.md References • deployment/upgrade.md • deployment/sr_operator.md • deployment/downgrade.md
g
@Rocky hmmm 
helm list -A | grep -i kube-starrocks
still says 
kube-starrocks-1.11.3
and 
3.5-latest
even though i'm definitely on 
4.0.2
and tried to update to 
1.11.4
r
Sorry, I encountered an error while processing your request: 429 RESOURCE_EXHAUSTED. {'error': {'code': 429, 'message': 'Resource has been exhausted (e.g. check quota).', 'status': 'RESOURCE_EXHAUSTED'}}
Open in Slack
PreviousNext
Powered by