http://starrocks.io logo
Join Slack
Powered by
Hi <@U0A71G31CDV> I want to restore the starrocks...
# questions-and-troubleshooting
s
Hi @Rocky I want to restore the starrocks cluster which is deployed using starrocks kubernetes operator using automated cluster snapshot in another plain cluster. Share any documentation or steps to achieve this
r
To restore a StarRocks cluster (shared-data mode) from an automated cluster snapshot into a new cluster using the StarRocks Kubernetes Operator, you should follow the Disaster Recovery (DR) workflow. This feature is supported starting from StarRocks v3.4.2 (with significant enhancements in v3.5 and v4.0) and requires StarRocks Operator v1.10.0 or later. Overview of Cluster Snapshot In a shared-data architecture, data is stored in object storage, but FE metadata is local. The Cluster Snapshot feature ensures a consistent point-in-time copy of both metadata and data references is stored in your object storage. ────────── Step 1: Prepare the Snapshot in the Source Cluster Ensure automated snapshots are enabled in your original cluster:
Copy code
sql
-- Enable automated snapshots
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON [STORAGE VOLUME <volume_name>];

-- Verify the snapshot status and retrieve the path
SELECT * FROM information_schema.cluster_snapshot_jobs;
Note the 
cluster_snapshot_path
(e.g., `s3://my-bucket/sr/meta/image/automated_cluster_snapshot_123456789`). Step 2: Prepare 
cluster_snapshot.yaml
for the New Cluster Create a configuration file named 
cluster_snapshot.yaml
that defines where the target cluster should pull the snapshot from. Template:
Copy code
yaml
cluster_snapshot:
    # The URI obtained from the source cluster
    cluster_snapshot_path: "<s3://my-bucket/sr/meta/image/automated_cluster_snapshot_123456789>"
    # Must match the volume name in the original cluster
    storage_volume_name: "my_s3_volume"

# Define the storage volume for the new cluster
storage_volumes:
  - name: "my_s3_volume"
    type: S3
    location: "<s3://my-bucket/sr/>"
    properties:
      - key: "aws.s3.region"
        value: "us-east-1"
      - key: "aws.s3.endpoint"
        value: "<https://s3.us-east-1.amazonaws.com>"
      - key: "aws.s3.access_key"
        value: "your_ak"
      - key: "aws.s3.secret_key"
        value: "your_sk"
Step 3: Create a Kubernetes ConfigMap Create a ConfigMap in your "plain" K8s cluster to hold the snapshot configuration:
Copy code
bash
kubectl create configmap cluster-snapshot-conf --from-file=cluster_snapshot.yaml=./cluster_snapshot.yaml -n starrocks
Step 4: Deploy the New Cluster via Operator Define your new cluster using the 
StarRocksCluster
CRD. You must include the 
disasterRecovery
field and mount the ConfigMap to the FE's 
conf
directory. Example StarRocksCluster YAML:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-restored
spec:
  # Enable disaster recovery mode
  disasterRecovery:
    enabled: true
  
  starRocksFeSpec:
    replicas: 3
    image: starrocks/fe-ubuntu:v3.5.0
    # Mount the config file to /opt/starrocks/fe/conf/cluster_snapshot.yaml
    configMaps:
      - name: cluster-snapshot-conf
        mountPath: /opt/starrocks/fe/conf
    # The Operator v1.10.0+ will automatically detect disasterRecovery 
    # and start the leader FE with the --cluster_snapshot flag.
  
  starRocksCnSpec:
    replicas: 3
    image: starrocks/cn-ubuntu:v3.5.0
Important Considerations: 1. Operator Version: Ensure you are using Operator v1.10.0 or higher. Older versions do not recognize the 
disasterRecovery
spec and will fail to start the FE with the necessary 
--cluster_snapshot
flag. 2. Shared-Data Only: This specific cluster-wide automated snapshot mechanism is only available for Shared-Data (Cloud-Native) clusters. For Shared-Nothing clusters, you must use the standard 
BACKUP
and 
RESTORE
commands via a repository. 3. Clean State: The Operator handles the cleaning of metadata and storage directories automatically during the DR recovery phase when 
disasterRecovery.enabled
is set to 
true
. 4. Finalization: Once the cluster is successfully restored and running, you should set 
disasterRecovery.enabled: false
in your CR to prevent the cluster from attempting to restore from the snapshot again on next restart.
s
@Rocky after using the above config iam getting error as grep: /opt/starrocks/fe/conf/fe.conf: No such file or directory It is expecting fe.conf need to be present, I am trying to update the fe.conf using config map reference in the example still is not taking https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/examples/starrocks/deploy_a_starrocks_cluster_running_in_shared_data_mode.yaml
r
The error 
grep: /opt/starrocks/fe/conf/fe.conf: No such file or directory
occurs because when you mount a Kubernetes ConfigMap directly to a directory (like 
/opt/starrocks/fe/conf
), Kubernetes overwrites the entire directory with the contents of the ConfigMap. This hides the default 
fe.conf
file bundled in the StarRocks container image, which the startup script requires. To fix this, you must use a 
subPath
 mount to add the 
cluster_snapshot.yaml
file into the existing configuration directory without deleting the other files. Revised Steps to Restore 1. Update your StarRocksCluster YAML Update the 
starRocksFeSpec
section of your CRD to use 
subPath
. This ensures only the snapshot file is injected.
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-restored
spec:
  # 1. Enable Disaster Recovery mode
  disasterRecovery:
    enabled: true
  
  starRocksFeSpec:
    replicas: 1 # Start with 1 FE for the initial recovery
    image: starrocks/fe-ubuntu:v3.5.0 # Ensure version is 3.4.2+
    configMaps:
      - name: cluster-snapshot-conf
        # Use the full path to the file
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        # Reference the specific key in your ConfigMap
        subPath: cluster_snapshot.yaml
    
    # 2. If you need custom fe.conf settings, add them here
    # Do NOT mount fe.conf via ConfigMap unless you include the entire file content
    config: |
      http_port = 8030
      query_port = 9030
2. Ensure your ConfigMap is correct The ConfigMap should contain the 
cluster_snapshot.yaml
key.
Copy code
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: cluster-snapshot-conf
data:
  cluster_snapshot.yaml: |
    cluster_snapshot:
      cluster_snapshot_path: "<s3://your-bucket/path/to/snapshot>"
      storage_volume_name: "your_volume_name"
    storage_volumes:
      - name: "your_volume_name"
        type: S3
        location: "<s3://your-bucket/path/>"
        properties:
          - key: "aws.s3.region"
            value: "us-east-1"
          # ... other storage properties
Key Technical Details 1. Why 
subPath
? In Kubernetes, 
mountPath: /opt/starrocks/fe/conf
replaces the folder. Using 
mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
with 
subPath: cluster_snapshot.yaml
tells Kubernetes to only place that single file into the existing directory. 2. Operator Behavior: When 
disasterRecovery.enabled: true
is set, the StarRocks Operator automatically appends the 
--cluster_snapshot
flag to the FE startup command. 3. Automatic Metadata Cleanup: The Operator (v1.10.0+) is designed to handle the cleanup of local metadata directories before starting the FE in recovery mode, so you don't need to manually delete PVCs if this is a fresh cluster deployment. 4. Post-Recovery: Once the cluster status shows 
Running
and your data is verified: * Update the CRD to set 
disasterRecovery.enabled: false
. * Scale up your FE/CN replicas to your desired production count. Troubleshooting the 
fe.conf
Error If you still see the error, it means you have another mount or configuration overriding the 
/opt/starrocks/fe/conf
directory. Ensure you don't have a 
volumeMounts
section in the 
starRocksFeSpec
that is mounting a volume to that same directory path. Use the 
config
field in the StarRocksCluster spec to manage FE parameters instead of mounting a separate 
fe.conf
file. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • administration/management/FE_configuration.md
s
@Rocky after incorporating the above changes my pod is in 0/1 state and I have mentioned 2 replica but it give up only 1 pod Also share the details document to restore the cluster in kubernetes
r
The behavior you are seeing—where the cluster remains at 1 Pod and is in a *0/1 state*—is actually the expected behavior during the initial phase of a Cluster Snapshot restoration using the StarRocks Operator. Here is the explanation for your observations and the complete step-by-step documentation to successfully restore your cluster. Why only 1 Pod? (Sequential Recovery) In a Disaster Recovery (DR) scenario, StarRocks must restore the Leader FE first. The Operator intentionally scales the FE StatefulSet to 1 replica regardless of your 
replicas: 2
setting. It does this to ensure only one node attempts to initialize the metadata from the snapshot. Once the first FE successfully restores the metadata and reaches a "Ready" state, the Operator will automatically: 1. Update the status to 
done
. 2. Scale up the remaining FE replicas (e.g., the 2nd pod). 3. Start the CN (Compute Node) pods. Why is the Pod 0/1 (Not Ready)? The FE pod will stay in a 
0/1
state while it is downloading and loading the snapshot metadata from your object storage (S3/GCS/Azure). The health checks (Readiness probes) will fail until the FE has fully loaded the metadata and opened its query ports. To monitor progress: Check the logs of the running FE pod:
Copy code
bash
kubectl logs -f <fe-pod-name> -n <namespace>
Look for lines like: 
successfully restore from cluster snapshot
or any errors related to S3 connectivity. ────────── Complete Documentation: Restoring via StarRocks Operator 1. Prerequisites * StarRocks Operator: v1.10.0 or higher. * StarRocks Version: v3.4.2 or higher. * Environment: A "plain" (new) Kubernetes cluster or a clean namespace. 2. Cleanup (If reusing an existing namespace) If you are not using a completely fresh cluster, you must clean up previous metadata to avoid conflicts: 1. Scale your StarRocksCluster CRD FE/CN replicas to 0. 2. Delete the existing FE and CN PersistentVolumeClaims (PVCs): 
kubectl delete pvc -l "<http://app.kubernetes.io/name=starrocks|app.kubernetes.io/name=starrocks>"
3. Create the Snapshot Configuration Create a file named 
cluster_snapshot.yaml
. Important: The 
storage_volume_name
must match the name used in your source cluster.
Copy code
yaml
cluster_snapshot:
    # URI from the source cluster (found in information_schema.cluster_snapshot_jobs)
    cluster_snapshot_path: "<s3://my-bucket/sr/meta/image/automated_cluster_snapshot_123456>"
    storage_volume_name: "my_s3_volume"

storage_volumes:
  - name: "my_s3_volume"
    type: S3
    location: "<s3://my-bucket/sr/>"
    properties:
      - key: "aws.s3.region"
        value: "us-east-1"
      - key: "aws.s3.endpoint"
        value: "<https://s3.us-east-1.amazonaws.com>"
      - key: "aws.s3.access_key"
        value: "your_ak"
      - key: "aws.s3.secret_key"
        value: "your_sk"
4. Create the ConfigMap
Copy code
bash
kubectl create configmap cluster-snapshot-conf --from-file=cluster_snapshot.yaml=./cluster_snapshot.yaml -n <namespace>
5. Deploy the Restored Cluster Apply your 
StarRocksCluster
CRD with the 
disasterRecovery
field enabled and the 
subPath
mount to ensure 
fe.conf
is not overwritten.
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-restored
spec:
  disasterRecovery:
    enabled: true  # This triggers the recovery mode logic
  
  starRocksFeSpec:
    replicas: 2
    image: starrocks/fe-ubuntu:v3.5.0
    configMaps:
      - name: cluster-snapshot-conf
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml  # Crucial: prevents overwriting the whole /conf directory
    # ... other FE specs
  
  starRocksCnSpec:
    replicas: 2
    image: starrocks/cn-ubuntu:v3.5.0
6. Verify and Finalize 1. Check CRD Status:
Copy code
bash
    kubectl get src starrocks-restored -o yaml
Look for 
status.disasterRecoveryStatus.phase
. It will move from 
todo
-> 
doing
-> 
done
.
2. Completion: Once the phase is 
done
, the Operator will start the 2nd FE and all CNs. 3. Disable DR Mode: To prevent the cluster from attempting to restore again on a future restart, update your YAML to set 
disasterRecovery.enabled: false
. Troubleshooting 
0/1
hangs: * Check Credentials: Ensure the S3 keys in 
cluster_snapshot.yaml
have permissions to read the path. * Check Path: Ensure the 
cluster_snapshot_path
points to the specific snapshot folder (the one containing the 
image
file), not just the root bucket. References • administration/cluster_snapshot.md • deployment/shared_data/feature-support-shared-data.md • administration/management/Backup_and_restore.md • deployment/shared_data/s3.md
s
@Rocky Instead of downloading the snapshot from s3, I have observed that it is created a new sub folder in s3 what I need to do here now
r
The observation that a new sub-folder is being created in S3 instead of downloading the snapshot indicates that your FE (Front End) node has started as a fresh cluster initialization rather than a restoration. In StarRocks shared-data mode, if the FE does not find valid metadata or a specific restore instruction, it generates a new Cluster ID (UUID) and creates a corresponding folder in your S3 bucket. To fix this and ensure the restore actually happens, follow these troubleshooting and correction steps: 1. Verify if the 
--cluster_snapshot
flag is active The StarRocks Operator must pass the 
--cluster_snapshot
flag to the FE binary to trigger the recovery logic. * Action: Run 
kubectl get pod <fe-pod-name> -o yaml
and look for the 
containers.args
or 
command
section. * Expected: You should see 
--cluster_snapshot
in the startup arguments. * If missing: Ensure your 
StarRocksCluster
CR has 
disasterRecovery.enabled: true
. If it is already 
true
, check your Operator version; you need Operator v1.10.0+. 2. Verify the 
cluster_snapshot.yaml
file inside the Pod Even if the flag is present, the FE will ignore it if it cannot find the configuration file or if the file is empty/malformed. * Action: Check if the file exists and has content:
Copy code
bash
    kubectl exec <fe-pod-name> -- cat /opt/starrocks/fe/conf/cluster_snapshot.yaml
* If missing: Your ConfigMap mount or 
subPath
is likely incorrect. Ensure the 
mountPath
is exactly 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
. 3. Check FE Logs for Restoration Errors The FE logs will explicitly state why it is skipping the restore or if it failed to connect to S3. * Action: 
kubectl logs <fe-pod-name> | grep -i "snapshot"
* Key Log Messages to look for: * `successfully restore from cluster snapshot`: Success. * `Failed to load cluster_snapshot.yaml`: Indicates the file is missing or has a syntax error. * `storage volume name does not match`: The 
storage_volume_name
in your YAML must be identical to the one used in the source cluster (often 
builtin_storage_volume
if you didn't name it). 4. Ensure Metadata PVCs are Empty If the FE pod restarts and finds any existing metadata in its local storage (from a previous failed attempt), it will attempt to start normally and skip the restoration flag. * Corrective Step: 1. Set 
replicas: 0
for FE and CN in your YAML. 2. Delete the FE PVCs: 
kubectl delete pvc -l "<http://app.kubernetes.io/name=starrocks|app.kubernetes.io/name=starrocks>"
. 3. Set 
replicas: 1
(for the leader) and apply the 
disasterRecovery: enabled: true
config again. 5. Correct 
cluster_snapshot.yaml
Structure Ensure your YAML follows this exact structure. Note that the 
cluster_snapshot_path
should point to the specific timestamped snapshot folder.
Copy code
yaml
cluster_snapshot:
    # Must point to the 'automated_cluster_snapshot_...' folder
    cluster_snapshot_path: "<s3://my-bucket/sr/meta/image/automated_cluster_snapshot_1734567890>"
    # MUST match the volume name in the source cluster
    storage_volume_name: "builtin_storage_volume" 

storage_volumes:
  - name: "builtin_storage_volume"
    type: S3
    location: "<s3://my-bucket/sr/>" # The root location
    properties:
      - key: "aws.s3.region"
        value: "us-east-1"
      - key: "aws.s3.access_key"
        value: "AKIA..."
      - key: "aws.s3.secret_key"
        value: "SECRET..."
Summary of what to do now: 1. Scale down the cluster to 0. 2. Delete the FE PVCs to ensure a clean slate. 3. Double-check the 
storage_volume_name
 in your source cluster (run 
SHOW STORAGE VOLUMES
in the old cluster). 4. Re-apply the CR with 
disasterRecovery.enabled: true
. 5. Watch the logs of the first FE pod immediately. It should show a download progress for the metadata image. References • administration/cluster_snapshot.md
s
@Rocky I have validated the snapshot it is valid but still not recognised, I am not able to restore because of this
r
If your StarRocks FE is creating a new sub-folder in S3 instead of downloading the snapshot, it means the FE process has started a new cluster initialization and completely skipped the restoration logic. This happens when the FE does not detect the 
--cluster_snapshot
flag or the 
cluster_snapshot.yaml
file is not being read correctly. To force the cluster to recognize the snapshot, follow these 3 critical steps: 1. Increase the 
generation
in your CRD The StarRocks Operator uses a 
generation
field to track disaster recovery attempts. If you are re-trying a restore, you must increment this number. Otherwise, the Operator might assume the DR process is already finished. Update your 
StarRocksCluster
YAML:
Copy code
yaml
spec:
  disasterRecovery:
    enabled: true
    generation: 2  # Increment this number (e.g., 1 -> 2) to trigger a fresh DR attempt
2. Verify the Pod's Startup Arguments Even with 
disasterRecovery: true
, you must confirm the Operator is actually passing the correct flag to the FE binary. * Run this command: 
kubectl get pod <fe-pod-name> -o jsonpath='{.spec.containers[0].args}'
* Look for: The string 
--cluster_snapshot
. * If missing: Your StarRocks Operator version may be below v1.10.0. If you cannot upgrade the operator, you can try adding the argument manually in the 
starRocksFeSpec
using the 
args
field (though this is not recommended as the operator usually manages it). 3. Ensure a "Clean Slate" (Delete PVCs) If the FE pod restarts and finds any existing files in its 
/opt/starrocks/fe/meta
directory (from the failed attempt where it created the new S3 folder), it will ignore the restoration flag on the next boot and just resume as that "new" cluster. You must perform this exact sequence: 1. Scale the cluster down: 
kubectl patch src <cluster-name> --type='merge' -p '{"spec":{"starRocksFeSpec":{"replicas":0}, "starRocksCnSpec":{"replicas":0}}}'
. 2. Wait for pods to disappear. 3. Delete all FE PVCs: 
kubectl delete pvc -l "<http://app.kubernetes.io/name=starrocks|app.kubernetes.io/name=starrocks>"
. 4. Update your YAML with the incremented 
generation: 2
and 
disasterRecovery: true
. 5. Re-apply the YAML. 4. Checklist for 
cluster_snapshot.yaml
 If it still fails, check the FE logs (
kubectl logs <fe-pod-name>
) for these specific errors: * "Fail to load cluster_snapshot.yaml": This means the 
subPath
mount failed or the path 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
is wrong. * "storage volume name does not match": The 
storage_volume_name
in your YAML must be identical to the name used in the original cluster (e.g., 
builtin_storage_volume
). * "snapshot path is invalid": Ensure 
cluster_snapshot_path
points to the folder containing the 
image
file (e.g., `s3://bucket/sr/meta/image/automated_cluster_snapshot_12345`). Summary of Documentation for K8s Restore The standard procedure for Kubernetes restoration is: 1. Prepare: Identify the snapshot URI and the original 
storage_volume_name
. 2. Clean: Delete existing PVCs in the target namespace. 3. Config: Create a ConfigMap with the 
cluster_snapshot.yaml
(using the template in the official docs). 4. Deploy: Apply the 
StarRocksCluster
CRD with 
disasterRecovery.enabled: true
and 
generation: 1
. 5. Monitor: Check the Operator status: 
kubectl get src <name> -o jsonpath='{.status.disasterRecoveryStatus}'
. It should progress from 
todo
-> 
doing
-> 
done
. 6. Finalize: Once 
done
, set 
disasterRecovery.enabled: false
to allow normal operation. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • deployment/deploy_manually.md
s
@Rocky please let me know can we connect over a call to debug the issue
r
As a chatbot named Rocky, I don’t have the ability to jump on a live call or screen-share. However, the StarRocks community and ecosystem have several ways for you to get more direct, hands-on help from experts: *1. Join the StarRocks Slack (Recommended)* This is the fastest way to interact with StarRocks engineers and community members in real-time. You can share your logs, YAML files, and screenshots there. * Link: starrocks.io/slack * Channels to use: 
#questions-and-troubleshooting
or 
#starrocks-operator
*2. StarRocks Community Forum* For deeper technical issues that might take time to investigate, the forum is a great place to post all your configuration details and logs. * Link: forum.starrocks.io *3. Enterprise Support (For 24/7 & Call Support)* If this is a production-critical issue and you require guaranteed response times or direct consultation (including calls), you may want to look into enterprise support from partners like CelerData or other regional vendors. They provide SLA-backed support which includes direct access to senior engineers via support tickets and calls. ────────── What to provide when you reach out on Slack: To help someone debug this quickly on Slack, please have the following ready: 1. Your StarRocksCluster YAML: (Specifically the 
starRocksFeSpec
and 
disasterRecovery
sections). 2. The FE Pod Logs: Run 
kubectl logs <fe-pod-name>
and look for the first 100 lines where the process starts. 3. The Operator Logs: Run 
kubectl logs -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator> -n <operator-namespace>
. This will tell us if the Operator is successfully "injecting" the restoration flag into your pod. 4. StarRocks Version: (e.g., v3.3.5) and Operator Version (e.g., v1.10.2). By providing these details in the 
#questions-and-troubleshooting
channel on Slack, you will likely get a response very quickly!
Open in Slack
PreviousNext
Powered by