http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> How to recover a FE node from a sna...
# questions-and-troubleshooting
d
@Rocky How to recover a FE node from a snapshot stored in s3?
r
To recover an FE node from a cluster snapshot stored in S3, you should use the Cluster Snapshot feature. This is the primary method for restoring the entire cluster state (including FE metadata) from a remote object store like S3. Prerequisites * You must have a cluster snapshot already created and stored in your S3 bucket. * You need the URI of the snapshot (e.g., `s3://my-bucket/snapshots/snapshot_name`). Step-by-Step Recovery Process 1. Configure the Recovery Metadata On the FE node that will act as the new Leader, you need to create or modify the 
fe/conf/cluster_snapshot.yaml
file to point to the S3 location. Example `cluster_snapshot.yaml`:
Copy code
yaml
cluster_snapshot:
    # The URI of the snapshot in S3
    cluster_snapshot_path: <s3://your-bucket-name/path/to/snapshot/>
    # The name of the storage volume. Must match the one used during backup.
    storage_volume_name: my_s3_volume

# Information of the storage volume in the new cluster
storage_volumes:
  - name: my_s3_volume
    type: S3
    location: <s3://your-bucket-name/path/to/snapshot/>
    properties:
      - key: aws.s3.region
        value: us-west-2
      - key: aws.s3.endpoint
        value: <https://s3.us-west-2.amazonaws.com>
      - key: aws.s3.access_key
        value: <YOUR_ACCESS_KEY>
      - key: aws.s3.secret_key
        value: <YOUR_SECRET_KEY>
2. Start the Leader FE Node Start the first FE node using the 
--cluster_snapshot
flag. This tells the FE to initialize its metadata from the snapshot defined in the YAML file.
Copy code
bash
./fe/bin/start_fe.sh --cluster_snapshot --daemon
3. Verify the Leader Node Log into the FE via MySQL client and check the status:
Copy code
sql
SHOW FRONTENDS;
Ensure the node is 
Alive
and its 
Role
is 
LEADER
. 4. Add and Start Other FE Nodes (Followers/Observers) If you have additional FE nodes to recover: 1. Clean the 
meta
directory on the other FE nodes to ensure they start fresh. 2. Start them using the 
--helper
flag pointing to the Leader FE you just started:
Copy code
bash
    ./fe/bin/start_fe.sh --helper <leader_ip>:<leader_edit_log_port> --daemon
Important Notes * Cross-Cluster Recovery: If you are restoring to a different cluster (different IPs), you must specify the new node information in the 
frontends
and 
compute_nodes
sections of the 
cluster_snapshot.yaml
file. * Metadata Reset: For manual metadata recovery from a local backup (if you manually downloaded the S3 snapshot files to the local disk), you would typically use the 
bdbje_reset_election_group = true
configuration in 
fe.conf
, but the 
--cluster_snapshot
method is the official automated way for S3-stored snapshots. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • deployment/shared_data/s3.md
d
@Rocky I have the following storage volume on my fe node
Copy code
storageVolumes:
      - name: fe-meta
        storageClassName: gp3
        storageSize: 100Gi
        mountPath: /opt/starrocks/fe/meta
I can see a snapshot directory in s3 like, 
<s3://starrocks_poc_starrocks/ae972000-11d8-4116-a218-38bc6d9a7492/meta/image/automated_cluster_snapshot_1774867715793/>
. Does that affect the recovery process since it doesn't match exactly what you described.
r
The directory structure you see in S3 is perfectly normal and expected for the Automated Cluster Snapshot feature in StarRocks (available in version 3.1 and later). The path you observed (
.../meta/image/automated_cluster_snapshot_1774867715793/
) follows the standard naming convention for this feature: 
{storage_volume_location}/{service_id}/meta/image/automated_cluster_snapshot_{timestamp}
. Why this path might look different If you were looking at documentation for traditional metadata backups (like using 
BACKUP
and 
RESTORE
commands) or simple manual file copies, you might have expected a different structure. However, the Automated Cluster Snapshot is a specific disaster recovery feature that: 1. Automatically uploads a snapshot of your FE metadata to S3 every time a checkpoint is created (controlled by 
automated_cluster_snapshot_interval_seconds
). 2. Coordinates metadata and data (on BE/CN) so you can restore the entire cluster to a specific point in time. How this affects the Recovery Process It doesn't "break" the recovery; it actually simplifies it if you are doing a full cluster restoration. Here is how it relates to your setup: 1. Local Storage (
fe-meta
): Your local 
mountPath: /opt/starrocks/fe/meta
is where the active FE metadata (BDB logs and current image) lives during runtime. This is the "hot" data. 2. S3 Snapshot: The path in S3 is the "cold" backup. To recover using this S3 snapshot: If your FE local metadata is lost or you are migrating to a new cluster, you perform a Cluster Restore: 1. Preparation: If you are restoring to a new cluster, you need to create a 
fe/conf/cluster_snapshot.yaml
file on your leader FE. In that file, you point the 
cluster_snapshot_path
specifically to that S3 URI:
Copy code
yaml
    cluster_snapshot:
        cluster_snapshot_path: <s3://starrocks_poc_starrocks/ae972000-11d8-4116-a218-38bc6d9a7492/meta/image/automated_cluster_snapshot_1774867715793/>
        storage_volume_name: <your_storage_volume_name>
2. Start FE: Start the leader FE with the recovery flag:
Copy code
bash
    ./fe/bin/start_fe.sh --cluster_snapshot --daemon
The FE will then automatically pull the metadata from that S3 path, initialize the local directory (
/opt/starrocks/fe/meta
), and reconstruct the cluster state. Summary * Is it correct? Yes. * Does it match? It matches the "Automated Cluster Snapshot" workflow, which is different from the manual "Metadata Backup" workflow. * Impact? It makes recovery easier because you have a system-generated, consistent snapshot of your metadata ready in S3. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • administration/management/FE_configuration.md
b
@Rocky How do you do this on the Celerdata hosted bring your own cloud version?
r
On the CelerData hosted Bring Your Own Cloud (BYOC) version, the core StarRocks engine is the same, but the management and infrastructure layers are handled through the CelerData Cloud Console. Depending on what you are trying to do ("this"), here is how you approach it on the BYOC platform: 1. Management & Infrastructure (CelerData Console) If "this" refers to cluster lifecycle management, you perform these actions in the CelerData Cloud Console (the web UI) rather than using command-line tools like 
sr-ctl
or manual configuration files: * Creating/Scaling Clusters: You don't need to manually add nodes. You use the "Clusters" tab in the console to create a new cluster or scale your Compute Nodes (CN) up/down. * Stopping/Starting: Clusters can be suspended or resumed from the UI to save costs. * Network & Security: Setting up VPC peering, PrivateLink, or IP allowlists is done in the "Network" or "Security" settings of the console. 2. SQL Operations (MySQL Client) If "this" refers to creating tables, running queries, or managing users, it is identical to open-source StarRocks. 1. Get the Endpoint: In the CelerData Console, click on your cluster to find the Host/Endpoint and the query port (default 
9030
). 2. Connect: Use any MySQL-compatible client:
Copy code
bash
    mysql -h <celerdata-cluster-endpoint> -P 9030 -u admin -p
3. Execute SQL: All standard StarRocks SQL (DDL, DML) works the same way. 3. Storage & Shared-Data Architecture CelerData BYOC clusters use the shared-data architecture (storage and compute separation) by default. * Storage Volumes: When you provision a BYOC cluster, CelerData automatically configures a Default Storage Volume pointing to the S3, GCS, or Azure Blob bucket in your own cloud account. * Creating Tables: You don't usually need to specify 
storage_volume
in your 
PROPERTIES
unless you want to use a non-default bucket.
Copy code
sql
    CREATE TABLE my_table (
        id INT,
        data VARCHAR(100)
    ) ENGINE=OLAP
    DUPLICATE KEY(id)
    DISTRIBUTED BY HASH(id);
    -- In BYOC, this automatically stores data in your cloud bucket.
4. Data Loading Because BYOC runs in your cloud environment, loading is often easier: * IAM Roles: You can grant the StarRocks cluster an IAM Role (AWS) or Service Account (GCP) during setup. This allows you to load data from your buckets using 
FILES()
or 
ROUTINE LOAD
without embedding secret keys in your SQL commands. * Example (Loading from S3):
Copy code
sql
    INSERT INTO my_table
    SELECT * FROM FILES(
        "path" = "<s3://my-bucket/data.parquet>",
        "format" = "parquet"
    );
Summary: * Infrastructure (Scaling, Upgrades, Monitoring): Use the CelerData Cloud UI. * Data Tasks (Tables, Queries, Loads): Use standard StarRocks SQL via your favorite IDE or terminal. References • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • deployment/shared_data/feature-support-shared-data.md • quick_start/shared-data.md
d
@Rocky Preparation: If you are restoring to a new cluster, you need to create a 
fe/conf/cluster_snapshot.yaml
file on your leader FE. In that file, you point the 
cluster_snapshot_path
specifically to that S3 URI:
Copy code
yaml
    cluster_snapshot:
        cluster_snapshot_path: <s3://starrocks_poc_starrocks/ae972000-11d8-4116-a218-38bc6d9a7492/meta/image/automated_cluster_snapshot_1774867715793/>
        storage_volume_name: <your_storage_volume_name>
2. Start FE: Start the leader FE with the recovery flag:
Copy code
bash
    ./fe/bin/start_fe.sh --cluster_snapshot --daemon
Can this be done using the kubernetes operator yaml/deployment?
r
Yes, you can perform a cluster restoration using the StarRocks Kubernetes Operator by translating the manual steps into Kubernetes configurations within your 
StarRocksCluster
Custom Resource (CR). Since you are restoring to a new cluster, follow these steps: 1. Create a ConfigMap for 
cluster_snapshot.yaml
 First, package your snapshot configuration into a Kubernetes ConfigMap. This allows the Operator to mount the file into the FE container.
Copy code
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fe-snapshot-config
  namespace: starrocks
data:
  cluster_snapshot.yaml: |
    cluster_snapshot:
        cluster_snapshot_path: <s3://starrocks_poc_starrocks/ae972000-11d8-4116-a218-38bc6d9a7492/meta/image/automated_cluster_snapshot_1774867715793/>
        storage_volume_name: <your_storage_volume_name>
    # Note: Define storage_volumes and nodes here if doing cross-cluster recovery 
    # as shown in the manual template.
2. Configure the 
StarRocksCluster
CR In your 
StarRocksCluster
YAML, you need to: 1. Mount the ConfigMap to the correct location (
/opt/starrocks/fe/conf/
). 2. Pass the 
--cluster_snapshot
flag to the FE startup command using the 
FE_ARGS
environment variable.
Copy code
yaml
apiVersion: <http://starrocks.com/v1init|starrocks.com/v1init>
kind: StarRocksCluster
metadata:
  name: starrockscluster-sample
spec:
  feSpec:
    replicas: 1 # Start with 1 replica (the Leader) for the restore process
    image: starrocks/fe-ubuntu:latest
    # Pass the startup flag via environment variables
    envVars:
      - name: FE_ARGS
        value: "--cluster_snapshot"
    # Mount the cluster_snapshot.yaml file
    storageVolumes:
      - name: snapshot-conf
        configMap:
          name: fe-snapshot-config
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml
3. Execution Process 1. Deploy the CR: Apply the 
StarRocksCluster
configuration. The Operator will start the FE with the recovery flag. 2. Monitor Logs: Check the FE logs (
kubectl logs <fe-pod-name>
) to ensure the restoration from the S3 URI is successful. 3. Scale Up: Once the Leader FE is restored and healthy, you can scale the 
replicas
in your 
feSpec
to your desired number (e.g., 3). The Operator will automatically handle adding the other FEs as followers. 4. Post-Restore: After the cluster is successfully restored and running, it is recommended to: * Remove the 
FE_ARGS: "--cluster_snapshot"
environment variable. * Remove the 
cluster_snapshot.yaml
volume mount. * Update the CR to persist the normal running state. Important Considerations: * Storage Volumes: Ensure that the 
storage_volume_name
referenced in your YAML exists or is defined in the 
storage_volumes
section of the 
cluster_snapshot.yaml
so the new cluster knows how to access the S3 bucket. * Permissions: Ensure the Kubernetes nodes or the FE Pod (via IAM Roles for Service Accounts - IRSA) have the necessary permissions to read from the S3 bucket URI. * Empty Meta Dir: Since this is a new cluster, the Persistent Volume Claims (PVCs) will be empty, satisfying the requirement to have a clean meta directory before restoration. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md
d
@Rocky I couldn't get that to work, it didn't seem to be picking up the 
--cluster_snapshot
argument. Furthermore, when using the kubernertes operator the syntax seems to be:
Copy code
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-poc
  namespace: starrocks-poc
spec:
  disasterRecovery:
    enabled: true
    generation: 1
From what I could see / get running (despite it not seemingly running disaster recovery / snapshot recovery). Are there examples of this working with the kubernetes operator?
r
The 
disasterRecovery
field in the 
StarRocksCluster
CRD is a specific feature designed to automate the Cluster Snapshot restoration process for shared-data clusters. If the operator is not picking up the configuration or injecting the 
--cluster_snapshot
argument, there are a few key things to verify: 1. Requirements for Cluster Snapshot The 
--cluster_snapshot
flag and the 
disasterRecovery
automation are only applicable to shared-data clusters (where 
run_mode = shared_data
). This feature is not used for traditional shared-nothing clusters. 2. The 
disasterRecovery
Specification In the StarRocks Operator, the 
disasterRecovery
field is typically used to trigger a restore from a snapshot stored in object storage. For the operator to successfully perform the restore, it usually requires a 
cluster_snapshot.yaml
file to be present or mapped. While the syntax you provided is correct for triggering the logic, the operator needs to know where to restore from. In a manual setup, this is done via 
fe/conf/cluster_snapshot.yaml
. In Kubernetes, you can provide this configuration via a 
ConfigMap
. Example StarRocksCluster with Disaster Recovery:
Copy code
yaml
apiVersion: <http://starrocks.com/v1|starrocks.com/v1>
kind: StarRocksCluster
metadata:
  name: starrocks-poc
spec:
  starRocksFeSpec:
    # Ensure you are using a version that supports Cluster Snapshot (v3.1+)
    image: starrocks/fe-ubuntu:3.4.2
    configMapInfo:
      # This ConfigMap should contain your 'cluster_snapshot.yaml'
      configMapName: fe-snapshot-config
      resolveKey: fe.conf
  disasterRecovery:
    enabled: true
    # Incrementing the generation triggers the operator to 
    # restart the FE with the --cluster_snapshot flag.
    generation: 1
3. Why 
--cluster_snapshot
might not be appearing If the operator is not adding the flag to the startup command: * Operator Version: Ensure you are using the latest version of the StarRocks Kubernetes Operator. This feature was introduced to handle the complex sequence of "Start Leader FE with flag -> Clean other FE/CN meta -> Start others as helpers." * Generation Field: The 
generation
field acts as a trigger. If the operator has already processed 
generation: 1
, it won't re-run the recovery logic unless you increment it to 
2
. * ConfigMap Mapping: The operator often expects the 
cluster_snapshot.yaml
to be provided in the same ConfigMap as your 
fe.conf
. 4. Manual Workaround via 
args
 If the automated 
disasterRecovery
field isn't behaving as expected in your environment, you can manually inject the argument into the FE spec to force the leader to start in snapshot mode:
Copy code
yaml
spec:
  starRocksFeSpec:
    # Use args to pass the flag directly if the automated field fails
    args:
      - --cluster_snapshot
Note: If you do this manually, you must be careful to remove the flag after the leader has successfully recovered and the metadata is restored, otherwise it will try to restore on every restart. Troubleshooting Steps: 1. Check Operator Logs: Look at the logs of the 
starrocks-controller
pod. It will specifically log if it detects the 
disasterRecovery
field and whether it is attempting to modify the FE startup command. 2. Verify Shared-Data: Confirm your 
fe.conf
(via ConfigMap) has 
run_mode = shared_data
. 3. Clean Meta: The manual process requires cleaning 
meta
directories on non-leader nodes. The operator handles this by recreating the pods, but ensure your Persistent Volume Claims (PVCs) aren't retaining old, corrupted state if the operator logic isn't fully clearing them. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md
d
@Rocky Can multiple 
configMapInfo
blocks be used or only one in the fe spec?
Copy code
configMapInfo:
      # This ConfigMap should contain your 'cluster_snapshot.yaml'
      configMapName: fe-snapshot-config
      resolveKey: fe.conf
r
In the StarRocks Kubernetes Operator (
StarRocksCluster
CRD), the 
configMapInfo
field in the 
fe
(and 
be
, 
cn
) spec is a single object, not an array. This means you can only specify one 
configMapInfo
block per component. Structure of 
configMapInfo
 It is defined as follows:
Copy code
yaml
starRocksFeSpec:
  configMapInfo:
    configMapName: fe-config
    resolveKey: fe.conf
Key details: * Only one ConfigMap: You can only link one ConfigMap via this specific field to act as the primary configuration source. * Purpose: The 
resolveKey
tells the operator which key inside that ConfigMap contains the actual configuration file (e.g., 
fe.conf
). * Handling multiple files: If you need to provide multiple configuration files (like 
fe.conf
plus custom scripts or additional XML files), you should include all of them as separate keys within the same ConfigMap. The Operator mounts the entire ConfigMap to the configuration directory of the pod. What if you need multiple ConfigMaps? If you absolutely require mounting files from different ConfigMaps, the standard 
configMapInfo
field won't support it directly as a list. You would typically need to: 1. Merge the data into a single ConfigMap. 2. Use a Helm chart override (if using the 
kube-starrocks
Helm chart) to add custom volumes and volume mounts to the FE pods via the 
podTemplate
or 
extraVolumes
fields if supported by your specific operator version. In summary: Only one 
configMapInfo
block is supported in the FE spec. References • deployment/sr_operator.md • administration/management/FE_configuration.md • deployment/shared_data/minio.md • quick_start/helm.md • deployment/shared_data/s3.md
d
@Rocky Provide an example of adding the FE conf and snapshot recovery conf. Show the config map and cluster configuration.
r
To add custom FE configurations and snapshot recovery settings in a Kubernetes environment using the StarRocks Operator, you need to define a 
ConfigMap
and reference it in your 
StarRocksCluster
resource. Below is an example showing how to configure 
fe.conf
and the 
cluster_snapshot.yaml
(used for cross-cluster or disaster recovery). 1. Create the ConfigMap This ConfigMap contains the main 
fe.conf
and the snapshot recovery configuration file 
cluster_snapshot.yaml
.
Copy code
yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: fe-config
  namespace: starrocks
data:
  fe.conf: |
    # Standard FE Configurations
    http_port = 8030
    rpc_port = 9020
    query_port = 9030
    edit_log_port = 9010
    
    # Enable metadata recovery flags if performing emergency recovery
    # start_with_incomplete_meta = true

  cluster_snapshot.yaml: |
    # Information for restoring from a cluster snapshot
    cluster_snapshot:
        # The URI of the snapshot in object storage
        cluster_snapshot_path: <s3://my-bucket/starrocks/snapshots/meta/image/automated_snapshot_12345>
        storage_volume_name: my_s3_volume

    storage_volumes:
      - name: my_s3_volume
        type: S3
        location: <s3://my-bucket/starrocks/>
        properties:
          - key: aws.s3.region
            value: us-west-2
          - key: aws.s3.endpoint
            value: <https://s3.us-west-2.amazonaws.com>
          - key: aws.s3.access_key
            value: <YOUR_ACCESS_KEY>
          - key: aws.s3.secret_key
            value: <YOUR_SECRET_KEY>
2. Configure the StarRocksCluster Resource In your 
StarRocksCluster
YAML, reference the ConfigMap under 
starRocksFeSpec
. To ensure the FE sees the 
cluster_snapshot.yaml
file, you can mount the ConfigMap as an additional volume.
Copy code
yaml
apiVersion: <http://starrocks.com/v1back1|starrocks.com/v1back1>
kind: StarRocksCluster
metadata:
  name: starrocks-cluster
  namespace: starrocks
spec:
  starRocksFeSpec:
    image: starrocks/fe-ubuntu:latest
    replicas: 3
    # Main FE configuration
    configMapInfo:
      configMapName: fe-config
      resolveKey: fe.conf
    # Mount the additional recovery config file
    storageVolumes:
      - name: recovery-config
        configMap:
          name: fe-config
          items:
            - key: cluster_snapshot.yaml
              path: cluster_snapshot.yaml
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml
Key Considerations for Snapshot Recovery: * Startup Flag: To trigger a restoration from the snapshot, the FE Leader must be started with the 
--cluster_snapshot
flag. In a Kubernetes environment, you may need to temporarily override the container 
args
in the 
StarRocksCluster
spec to include this flag. * Permissions: Ensure the FE pods have the necessary IAM roles or secret keys (as defined in the 
cluster_snapshot.yaml
) to access the object storage bucket. * Shared-Data Mode: Cluster snapshots are primarily designed for clusters running in shared-data mode (starting from v3.1). I'm Rocky, your StarRocks assistant! Let me know if you need help with the specific S3 or HDFS parameters for your recovery setup. References • administration/cluster_snapshot.md • deployment/sr_operator.md • quick_start/helm.md • administration/management/FE_configuration.md
d
@Rocky I don't see 
storageVolumes.configMap
showing up in the CRD. Where does this come from?
r
The 
storageVolumes
field in the StarRocks Kubernetes Operator (specifically within the 
StarRocksCluster
CRD) is used to mount additional volumes into the FE, BE, or CN pods. If you are looking for 
storageVolumes.configMap
and cannot see it, here is the breakdown of why that might be: 1. It is part of the 
storageVolumes
list In the 
StarRocksCluster
CRD (under 
starRocksFeSpec
, 
starRocksBeSpec
, or 
starRocksCnSpec
), the 
storageVolumes
field is an array. Each item in that array can define a 
configMap
. The structure generally looks like this:
Copy code
yaml
spec:
  starRocksBeSpec:
    storageVolumes:
      - name: my-custom-config
        mountPath: /etc/starrocks/custom
        configMap: # <--- This is the field you're looking for
          name: my-config-map-name
2. Check your Operator Version The support for mounting custom 
ConfigMaps
or 
Secrets
via the 
storageVolumes
field was added in later versions of the StarRocks Operator. if you are using an older version (e.g., earlier than v1.5.x), this field might not be available or might have a different schema. * You can verify your CRD definition by running:
Copy code
bash
    kubectl get crd <http://starrocksclusters.starrocks.com|starrocksclusters.starrocks.com> -o yaml | grep -A 20 storageVolumes
3. Difference between 
storageSpec
and 
storageVolumes
 It is common to confuse these two: * 
storageSpec
: Used specifically for the data and log persistent volumes (PVCs). It does not support 
configMap
because its purpose is persistent data storage. * 
storageVolumes
: Used for additional mounts, such as configuration files, scripts, or credentials. This is where 
configMap
, 
secret
, and 
hostPath
are defined. Where does it "come from"? It is defined in the StarRocks Operator API. It maps directly to the standard Kubernetes Volume source types. If you don't see it in your environment, you likely need to update your Operator's CRD manifests to the latest version. Note: If you are trying to inject custom configuration files (like 
fe.conf
or 
be.conf
), you typically use the 
configConfigMap
field (for the main config file) rather than 
storageVolumes
. 
storageVolumes
is better for secondary files like Kerberos keytabs or SSL certificates. References • quick_start/helm.md • administration/management/FE_configuration.md • deployment/shared_data/s3.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md
Open in Slack
PreviousNext
Powered by