http://starrocks.io logo
Join Slack
Powered by
<@U0A71G31CDV> if I want to restore the standalone...
# questions-and-troubleshooting
s
@Rocky if I want to restore the standalone server starrocks Cluster to eks what are steps
r
To restore a standalone StarRocks cluster to Amazon EKS, the recommended approach is to use the BACKUP and RESTORE mechanism with Amazon S3 as the intermediate storage. Phase 1: Prepare the Standalone Cluster You must first back up your data from the existing standalone server to an S3 bucket. 1. Create a Backup Repository: Connect to your standalone StarRocks cluster and create a repository pointing to an S3 bucket.
Copy code
sql
    CREATE REPOSITORY my_s3_repo
    WITH BROKER
    ON LOCATION "<s3a://your-bucket-name/starrocks-backup>"
    PROPERTIES(
        "aws.s3.access_key" = "YOUR_ACCESS_KEY",
        "aws.s3.secret_key" = "YOUR_SECRET_KEY",
        "aws.s3.region" = "us-east-1"
    );
Note: Use 
s3a://
protocol for the repository location. 2. Run the Backup: Back up the desired databases or tables.
Copy code
sql
    BACKUP SNAPSHOT your_db.snapshot_name
    TO my_s3_repo
    ON (your_table_name);
You can check the progress using 
SHOW BACKUP;
. Phase 2: Set up StarRocks on EKS 1. Deploy the StarRocks Operator: Use Helm to install the StarRocks Operator on your EKS cluster.
Copy code
bash
    helm repo add starrocks <https://starrocks.github.io/starrocks-kubernetes-operator>
    helm repo update
    helm install starrocks-operator starrocks/operator
2. Deploy the StarRocks Cluster: Create a 
StarRocksCluster
custom resource. Ensure the EKS nodes or pods have the necessary IAM permissions (or provide Access Keys) to access the S3 bucket.
Copy code
bash
    helm install starrocks-cluster starrocks/starrocks
Phase 3: Restore to EKS 1. Create the same Repository on EKS: Connect to the new StarRocks cluster on EKS (via the LoadBalancer or Port-forward) and create the same repository definition.
Copy code
sql
    CREATE REPOSITORY my_s3_repo
    WITH BROKER
    ON LOCATION "<s3a://your-bucket-name/starrocks-backup>"
    PROPERTIES(
        "aws.s3.access_key" = "YOUR_ACCESS_KEY",
        "aws.s3.secret_key" = "YOUR_SECRET_KEY",
        "aws.s3.region" = "us-east-1"
    );
2. Restore the Data: Run the restore command to pull the snapshot into the EKS cluster.
Copy code
sql
    RESTORE SNAPSHOT your_db.snapshot_name
    FROM my_s3_repo
    ON (your_table_name);
Monitor progress with 
SHOW RESTORE;
. Key Considerations * Version Compatibility: Ensure the StarRocks version on EKS is equal to or higher than the standalone version. * Broker: In K8s deployments, the StarRocks Operator typically handles the internal communication, but the 
WITH BROKER
syntax is still used in the SQL command for S3 repositories (the FE/BE acts as the broker if one isn't explicitly deployed). * Permissions: On EKS, it is best practice to use IAM Roles for Service Accounts (IRSA) instead of hardcoding Access Keys in the 
CREATE REPOSITORY
statement. If using IRSA, you can set 
"aws.s3.use_instance_profile" = "true"
. References • deployment/sr_operator.md • deployment/helm.md
s
@Rocky iwant to cluster snapshot yaml and disaster recovery approach in eks
r
To implement a disaster recovery (DR) approach and utilize cluster snapshots in Amazon EKS, the strategy depends on whether you are using a Shared-data (Cloud Native) or Shared-nothing (Standard) architecture. 1. Cluster Snapshot YAML (Shared-data Clusters) For shared-data clusters, StarRocks uses a feature called Cluster Snapshot for full disaster recovery (metadata + data). The 
cluster_snapshot.yaml
file is used during the recovery phase to define where the snapshot is located and how to map it to the new EKS cluster. cluster_snapshot.yaml Template This file is typically placed in the FE node's 
conf/
directory during recovery.
Copy code
yaml
cluster_snapshot:
    # URI of the snapshot in S3
    cluster_snapshot_path: <s3://your-bucket/starrocks/meta/image/automated_cluster_snapshot_1704038400000>
    # Must match the original cluster's storage volume name
    storage_volume_name: my_s3_volume

# [Optional] Map nodes in the new EKS cluster
frontends:
  - host: starrocks-fe-0.starrocks-fe-service.default.svc.cluster.local
    edit_log_port: 9010
    type: follower

compute_nodes:
  - host: starrocks-cn-0.starrocks-cn-service.default.svc.cluster.local
    heartbeat_service_port: 9050

storage_volumes:
  - name: my_s3_volume
    type: S3
    location: <s3://your-bucket/starrocks/>
    properties:
      - key: aws.s3.region
        value: us-east-1
      - key: aws.s3.use_instance_profile
        value: "true"
How to apply in EKS 1. Create a ConfigMap: Store the YAML in Kubernetes.
Copy code
bash
    kubectl create configmap sr-recovery-config --from-file=cluster_snapshot.yaml
2. Mount to StarRocksCluster: In your 
StarRocksCluster
CRD, mount this ConfigMap to the FE pods at 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
. 3. Start with Snapshot Flag: The Leader FE must be started with the 
--cluster_snapshot
flag. In the Operator, you may need to add this to the 
image.args
or via a custom startup script. ────────── 2. Disaster Recovery Approach in EKS Method A: Shared-data (Cloud Native) - Recommended for EKS * Backbone: All data is already in S3. * Backup: Enable automated metadata snapshots via SQL:
Copy code
sql
    ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME my_s3_volume;
* DR Recovery: 1. Deploy a new StarRocks cluster on EKS. 2. Provide the 
cluster_snapshot.yaml
(as shown above). 3. Restart FE nodes. The cluster will pull the full state (databases, tables, users, and data links) from S3. Method B: Shared-nothing (Standard) - SQL Based If you are moving from a standalone server (shared-nothing) to EKS, you use the standard Backup & Restore workflow: 1. Remote Repository: Create a repository pointing to S3 on both source (standalone) and target (EKS) clusters. 2. Backup: Run 
BACKUP SNAPSHOT db.snap1 TO repo ON (table1);
on the standalone server. 3. Cross-Region/Cluster: If the EKS cluster is in a different region, ensure S3 replication is active or use a global bucket. 4. Restore: Run 
RESTORE SNAPSHOT db.snap1 FROM repo ON (table1);
on the EKS cluster. Summary Comparison Feature | Shared-data (Snapshot) | Shared-nothing (Backup) Object Storage | Primary storage (S3) | Intermediate storage (S3) Recovery YAML | Required (
cluster_snapshot.yaml
) | Not required Recovery Scope | Entire Cluster (Metadata + Data) | Granular (DB or Table level) Automation | 
ADMIN SET AUTOMATED...
| Manual SQL or cron jobs Note: Shared-data clusters do not support the standard 
BACKUP
and 
RESTORE
SQL commands; they must use the Cluster Snapshot approach in-place S3 snapshots or the Cluster Snapshot feature. References • administration/management/Backup_and_restore.md • sql-reference/sql-statements/backup_restore/RESTORE.md
s
@Rocky in the cluster snapshot yaml storage volume refers to snapshot storage volume or data storage volume which we want to restore
r
In the 
cluster_snapshot.yaml
file, the 
storage_volume_name
refers to the shared storage volume used by the cluster to store both its data and its automated metadata snapshots. In StarRocks' shared-data (cloud-native) architecture, there is no distinction between a "snapshot volume" and a "data volume." Instead, the system continuously persists data to a defined Storage Volume (e.g., S3). When you enable automated snapshots, the metadata (FE state) is also saved to this same volume. How it works in the YAML: 1. 
cluster_snapshot.storage_volume_name
: This is the logical name of the volume as it was defined in the original (source) cluster (e.g., 
s3_volume
). It must match the original name exactly. 2. 
storage_volumes
(the list at the bottom): This is where you provide the physical connection details (S3 bucket path, region, and credentials) for that logical name so the new EKS cluster knows how to reach the data. Why they must match: When you restore to EKS, the metadata snapshot being loaded contains internal pointers that rely on that specific volume name. If the volume name in the new cluster doesn't match the one in the snapshot, the FE will fail to map the metadata to the actual data files in S3. Example snippet:
Copy code
yaml
cluster_snapshot:
    # Path to the specific metadata snapshot in S3
    cluster_snapshot_path: <s3://my-bucket/starrocks/meta/image/automated_snapshot_12345>
    # The name used in the SOURCE cluster
    storage_volume_name: my_s3_volume 

storage_volumes:
  - name: my_s3_volume  # Must be the same as above
    type: S3
    location: <s3://my-bucket/starrocks/> # The root path where data lives
    properties:
      - key: aws.s3.region
        value: us-east-1
Summary The 
storage_volume_name
refers to the Data Storage Volume that you are restoring. Because you are in a shared-data environment, this volume is also where the snapshot files are located. References • administration/cluster_snapshot.md • quick_start/shared-data.md • deployment/shared_data/feature-support-shared-data.md References • administration/cluster_snapshot.md • quick_start/shared-data.md • deployment/shared_data/feature-support-shared-data.md • deployment/shared_data/azure.md
s
@Rocky In my standalone server snapshot configured in one storage volume and data is in different storage volume
r
In a StarRocks shared-data architecture where you have separate storage volumes for your data and your snapshots, the 
cluster_snapshot.yaml
configuration must account for both. Here is how you handle this in your YAML file for the EKS restore: 1. 
cluster_snapshot
section This section specifically points the recovery process to the Metadata Snapshot. * 
cluster_snapshot_path
: This must be the URI pointing to the specific snapshot folder (e.g., `s3://snapshot-bucket/meta/image/automated_cluster_snapshot_...`). * 
storage_volume_name
: This refers to the volume name where the snapshot was stored in the original cluster. 2. 
storage_volumes
section You must list all storage volumes that the original cluster relied on. If your data lives in one volume and your snapshots in another, you must define both in the list. This ensures that once the metadata is restored, the cluster knows how to connect to the physical S3 locations for your actual table data. Example cluster_snapshot.yaml for Multiple Volumes
Copy code
yaml
cluster_snapshot:
    # URI of the snapshot files
    cluster_snapshot_path: <s3://snapshot-bucket/sr/meta/image/automated_cluster_snapshot_1710000000>
    # The volume name where the snapshot above is stored
    storage_volume_name: snapshot_volume 

frontends:
  - host: starrocks-fe-0.starrocks-fe-service.default.svc.cluster.local
    edit_log_port: 9010
    type: follower

compute_nodes:
  - host: starrocks-cn-0.starrocks-cn-service.default.svc.cluster.local
    heartbeat_service_port: 9050

storage_volumes:
  # 1. Define the Snapshot Volume
  - name: snapshot_volume
    type: S3
    location: <s3://snapshot-bucket/sr/>
    properties:
      - key: aws.s3.region
        value: us-east-1
      - key: aws.s3.use_instance_profile
        value: "true"

  # 2. Define the Data Volume (Crucial for reading your tables!)
  - name: data_volume
    type: S3
    location: <s3://data-bucket/sr-data/>
    properties:
      - key: aws.s3.region
        value: us-east-1
      - key: aws.s3.use_instance_profile
        value: "true"
Why this is necessary: * Metadata Recovery: The FE uses the 
storage_volume_name
under 
cluster_snapshot
to find and download the metadata image. * Data Linkage: Once the metadata is loaded, it contains references to tables. Those tables point to 
data_volume
. If 
data_volume
is not defined in the YAML, the cluster will not know the S3 credentials or path to access the actual data files, and queries will fail. Important Note for EKS: When restoring to EKS via the StarRocks Operator: 1. Mount the YAML: Use a ConfigMap to mount this file to 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
in your FE pods. 2. Clean Directories: Before starting the restore on EKS, ensure the FE metadata directory (
/opt/starrocks/fe/meta
) on your persistent volumes is empty. 3. Start Flag: The Leader FE must be started with the 
--cluster_snapshot
flag. You can usually achieve this by modifying the 
entrypoint
or 
command
in your Helm chart/Operator spec for the initial recovery run. References • administration/cluster_snapshot.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • deployment/shared_data/feature-support-shared-data.md • administration/management/Backup_and_restore.md References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md
2 Views
Open in Slack
PreviousNext
Powered by