< Rocky> how can I take backup of fe metadata incase if it g StarRocks #questions-and-troubleshooting

<@U0A71G31CDV> how can I take backup of fe metadat...

Mohan

09/02/2026, 6:14 AM

@Rocky how can I take backup of fe metadata incase if it gets corrupted or pvc deleted accidentally? I'm using 4.0.8 shared data architecture

Rocky

09/02/2026, 6:14 AM

In StarRocks 4.0.8 with shared-data architecture, backing up FE metadata is critical because while your table data lives in object storage, the "brain" of the cluster (catalogs, users, permissions, and data locations) lives in the FE's internal BDB JE metadata. To protect against PVC deletion or metadata corruption, you should use the following strategies: 1. Primary Method: Automated Cluster Snapshot Starting from version 3.4.2+, StarRocks supports Cluster Snapshots specifically for shared-data clusters. This feature automatically backs up the entire FE metadata state to your backend object storage (S3/GCS/Azure/MinIO). * Enable it:

Copy code

sql
    ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON [STORAGE VOLUME your_volume_name];

* How it works: Every time an FE completes a metadata checkpoint (default every 10 minutes), it automatically uploads a metadata image to the object store. * Why it's safe for PVC deletion: Since the snapshot is stored in the remote object storage, even if your Kubernetes PVC is accidentally deleted, you can restore the cluster state by pointing a new FE to that snapshot path. * Monitor jobs:

Copy code

sql
    SELECT * FROM information_schema.cluster_snapshot_jobs;

2. Manual Metadata Image (Checkpoint) If you want to manually trigger a point-in-time backup of the metadata to the local disk (the PVC) before a dangerous operation:

Copy code

sql
ALTER SYSTEM CREATE IMAGE;

This forces the FE to create a new

image.xxxx

file in the

meta/image

directory. 3. Kubernetes-Level Protection (PVC Backup) Since you are using PVCs, you should leverage Kubernetes native backup tools for the

fe-meta

volume: * Volume Snapshots (CSI): If your storage provider supports CSI snapshots, create a

VolumeSnapshot

resource for your FE metadata PVCs. * Velero: Use Velero to back up your entire StarRocks namespace, including the PVCs. This is the industry standard for "accidental PVC deletion" scenarios. 4. Recovery Procedure If your FE metadata is corrupted or the PVC is gone: 1. Clean the metadata directory (or provision a new PVC). 2. Restore from Cluster Snapshot: Start the Leader FE with the

--cluster_snapshot

flag.

Copy code

bash
    ./fe/bin/start_fe.sh --cluster_snapshot --daemon

_Note: For cross-cluster recovery, you may need to configure

fe/conf/cluster_snapshot.yaml

to point to the correct URI in object storage._ 3. Start other FEs using the

--helper

flag to synchronize from the restored Leader. Summary of Best Practices * Turn on Automated Cluster Snapshots immediately; it is the most robust way to recover a shared-data cluster. * Store metadata in a dedicated StorageClass with a

reclaimPolicy: Retain

so that even if the StarRocks deployment is deleted, the PVC remains. * Regularly verify your snapshots by checking

information_schema.cluster_snapshots

. References • administration/cluster_snapshot.md • administration/management/Backup_and_restore.md

Mohan

09/02/2026, 6:28 AM

@Rocky cluster snapshot automation gave this error > SELECT * FROM information_schema.cluster_snapshot_jobs\G; *************************** 1. row *************************** SNAPSHOT_NAME: automated_cluster_snapshot_1788330435590 JOB_ID: 83057 CREATED_TIME: 2026-09-02 115715 FINISHED_TIME: NULL STATE: ERROR DETAIL_INFO: ERROR_MESSAGE: upload image failed, err msg: java.lang.NullPointerException 1 row in set (0.01 sec)

Rocky

09/02/2026, 6:28 AM

The

java.lang.NullPointerException

during the upload image phase of a cluster snapshot usually indicates that the Frontend (FE) Leader is unable to resolve a valid storage location or credential object. Because Automated Cluster Snapshots (introduced in v3.4.2+) are still a relatively new feature, this specific NPE in version 4.0.8 is likely due to a configuration gap or a known path-resolution issue. Troubleshooting Steps 1. Verify your Storage Volume If you didn't specify a volume in the

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON

command, StarRocks attempts to use the Default storage volume. If no volume is marked as default, or if the default volume is misconfigured, it can trigger an NPE. * Check existing volumes:

Copy code

sql
    SHOW STORAGE VOLUMES;

* Verify the
IS_DEFAULT
column: Ensure at least one volume is marked as

true

. If not, set one:

Copy code

sql
    ALTER STORAGE VOLUME your_volume_name SET IS_DEFAULT = true;

* Try specifying the volume explicitly: Disable the current automation and re-enable it by explicitly naming the volume:

Copy code

sql
    ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF;
    ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME your_volume_name;

2. Check S3 Path-Style Access A known issue exists where the FE does not always correctly handle "Path Style" access (required for MinIO or some S3-compatible providers) during the image upload phase, even if it works for general data. * Ensure your Storage Volume properties include

"aws.s3.enable_path_style_access" = "true"

if you are using MinIO. 3. Inspect Detailed Logs The summary in

information_schema.cluster_snapshot_jobs

is often truncated. To find the root cause, you need the full Java stack trace: 1. Log into your Leader FE node. 2. Search

fe.warn.log

fe.log

for the string

NullPointerException

. 3. Pro Tip: If the log only shows a single line for the error, add this to your

fe/conf/fe.conf

and restart the FE to force a full stack trace:

Copy code

bash
    JAVA_OPTS="-XX:-OmitStackTraceInFastThrow"

4. Version Consideration There have been significant fixes to the Cluster Snapshot logic in the 4.1.x release branch (e.g., fixes for

CatalogRecycleBin

and snapshot integrity). If the troubleshooting above doesn't resolve it, consider upgrading to a later patch version of the 4.x series where these edge cases are better handled. Manual Backup (Workaround) While you fix the automated snapshot, you can manually verify your metadata is safe by creating a local image:

Copy code

sql
ALTER SYSTEM CREATE IMAGE;

This forces a checkpoint on the FE. You can then manually copy the

fe/meta/image/image.xxxx

file from the PVC to your object storage for safe keeping. References • administration/cluster_snapshot.md • sql-reference/sql-statements/cluster-management/storage_volume/SHOW_STORAGE_VOLUMES.md • Issue #61590: Cluster snapshot fails to upload to MinIO References • administration/cluster_snapshot.md • administration/management/BE_configuration.md • faq/Deploy_faq.md

Mohan

09/02/2026, 6:31 AM

@Rocky I've create volume with GS type GCS bucket and giving specific volume as well

Copy code

2026-09-02 11:57:37.013+05:30 WARN (cluster_snapshot_checkpoint_scheduler|180) [FileSystem.createFileSystemInternal():3749] Failed to initialize filesystem gs://<bucket>//d2b141e8-78d8-4279-b182-9c906378798d/meta/image/automated_cluster_snapshot_1788330435590: java.lang.NullPointerException

Rocky

09/02/2026, 6:31 AM

The

java.lang.NullPointerException

(NPE) during the upload image phase of a Cluster Snapshot typically occurs when the Frontend (FE) Leader fails to initialize the filesystem client for your GCS bucket. In version 4.0.8, this is most often caused by a URI construction error or missing mandatory properties in the

GS

type storage volume. Here is how to resolve this: 1. Avoid the Trailing Slash (Double Slash Issue) The error log shows a double slash:

gs://<bucket>//d2b141e8...

. If your Storage Volume

LOCATION

was defined with a trailing slash (e.g., `gs://my-bucket/`), the snapshot logic may be incorrectly appending another slash when building the internal path. Even though some StarRocks versions have fixes for this, GCS URIs are strictly validated. Fix: Recreate the storage volume without the trailing slash:

Copy code

sql
-- Use '<gs://bucket>' instead of '<gs://bucket/>'
ALTER STORAGE VOLUME your_volume_name SET LOCATIONS = ("<gs://your-bucket-name>");

2. Recommended: Switch to S3-Compatible Type While StarRocks supports

TYPE = GS

, the most stable path for FE metadata operations (like snapshots) on GCS is using the S3-compatible XML API with HMAC keys. This bypasses potential native GCS client initialization issues in the FE. Recreate your volume using this pattern:

Copy code

sql
CREATE STORAGE VOLUME gcs_snapshot_vol
TYPE = S3
LOCATIONS = ("<s3://your-bucket-name>")
PROPERTIES
(
    "enabled" = "true",
    "aws.s3.endpoint" = "<https://storage.googleapis.com>",
    "aws.s3.region" = "auto", -- or your specific region
    "aws.s3.access_key" = "<YOUR_GCS_HMAC_ACCESS_KEY>",
    "aws.s3.secret_key" = "<YOUR_GCS_HMAC_SECRET_KEY>",
    "aws.s3.enable_path_style_access" = "true"
);

-- Set as default and enable snapshot
SET gcs_snapshot_vol AS DEFAULT STORAGE VOLUME;
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME gcs_snapshot_vol;

3. Verify Mandatory
GS
Properties If you must use

TYPE = GS

(native), ensure you have provided the service account credentials correctly. An NPE often occurs if the code expects a Service Account JSON but finds a null reference. For

TYPE = GS

, you should use these properties (available in 3.5.1+ and 4.0.x): *

"gcp.gcs.use_compute_engine_service_account" = "true"

(if on GKE with Workload Identity/SA). * Or provide the explicit Service Account keys:

"gcp.gcs.service_account_email"

"gcp.gcs.service_account_private_key_id"

, and

"gcp.gcs.service_account_private_key"

. 4. Check FE Logs for the "Root Cause" The NPE in

information_schema

is just a summary. To see exactly which field is null, check the FE Leader log: 1. Go to the FE

log/fe.warn.log

. 2. Search for the timestamp

2026-09-02 11:57:37

. 3. Look for the full Java Stack Trace. It will likely point to a specific missing configuration key (e.g.,

GCS_CREDENTIALS

endpoint

). Summary Checklist 1. Check for trailing slash in

SHOW STORAGE VOLUMES

. If present, remove it. 2. Try the S3-compatible approach (Type = S3) as a workaround; it is generally more robust for metadata transfers in current 4.0.x versions. 3. Ensure your FE has network access to

<http://storage.googleapis.com|storage.googleapis.com>

from the node where the Leader FE is running. References • administration/cluster_snapshot.md • deployment/shared_data/gcs.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md References • administration/cluster_snapshot.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md

Rocky

09/02/2026, 6:31 AM

• deployment/shared_data/feature-support-shared-data.md

Mohan

09/02/2026, 7:04 AM

@Rocky what is bdb and other folder info in fe metadata and what is being stored in each one and how it is being stored, format of data etc..?

Rocky

09/02/2026, 7:05 AM

In StarRocks, the FE metadata directory (default:

fe/meta

) is the most critical part of your cluster. It stores the "brain" of the system, including table schemas, data distribution (tablets), user privileges, and load job states. Here is a breakdown of the subfolders and how data is stored: 1.
bdb/
(Berkeley DB Java Edition) * What is stored: The Edit Log (Journals). * Role: Every time you perform a metadata-changing operation (e.g.,

CREATE TABLE

LOAD

ALTER

), the Leader FE writes a record to this log. Follower FEs replay these logs to keep their memory in sync. * Format: Log-structured binary files (suffix

.jdb

). * Mechanism: StarRocks uses BDB JE as a high-performance consensus layer (similar to Raft/Paxos) to ensure that metadata changes are replicated across more than half of the Follower nodes before being committed. 2.
image/
* What is stored: Metadata Checkpoints and Node State. * Role: * Image files (
image.xxxx
): These are full "snapshots" of the entire metadata state at a specific log version. Instead of replaying millions of logs from the beginning of time, an FE loads the latest image into memory and then only replays the few logs generated after that image was created. * ROLE
file: A text file identifying if this node is a FOLLOWER or OBSERVER and its unique node name (e.g.,

10.x.x.x_9010_12345

). * VERSION
file: Contains the

cluster_id

(used to prevent nodes from joining the wrong cluster), the cluster

token

(for security), and the StarRocks version that last touched the metadata. * Format: The image file is a binary dump of Java objects (using custom serialization). It is not human-readable. 3.
small_files/
* What is stored: Files uploaded via the File Manager. * Role: Stores small external files like SSL certificates for external tables, dictionary files, or Kerberos keytabs. * Format: The raw files as they were uploaded. 4. How They Work Together (The Checkpoint Process) 1. Write: The Leader writes metadata changes to the

bdb/

logs. 2. Checkpoint: Periodically (by default every 10 minutes or 100,000 logs), a background thread (the Checkpointer) takes the current in-memory state, serializes it, and saves it as a new

image.xxxx

file in the

image/

.jdb

) | Incremental changes (DDL/DML logs). image/
| Checkpoint | Binary | Full point-in-time snapshot of the FE's brain. image/
| VERSION/ROLE | Text/Properties | Cluster ID, node role, and security tokens. small_files/
| Small Files | Raw | Certificates, keytabs, and user-uploaded files. Crucial Note for 4.0.8 Shared-Data: In shared-data architecture, your actual data (Parquet/ORC/Segment files) is in GCS, but the mapping of which tablets are in which files is stored in these FE metadata files. If you lose

fe/meta

, the system will not know how to find your data in GCS. Always ensure you have Automated Cluster Snapshots or PVC backups enabled as discussed earlier. References • administration/management/FE_configuration.md • introduction/Architecture.md • administration/Meta_recovery.md

Mohan

09/02/2026, 8:49 AM

@Rocky is there any retention on this table

information_schema.cluster_snapshot_jobs

? does it store all records continously? , I see only one snapshot data is stored in bucket

Rocky

09/02/2026, 8:50 AM

Regarding your observations about the Automated Cluster Snapshot retention: 1. Why is only one snapshot in the bucket? This is the intended behavior of the current implementation. According to the StarRocks documentation: * Single-Snapshot Policy: The system automatically maintains a snapshot that closely follows the latest cluster state. * Automatic Purge: Historical snapshots are dropped immediately after a new one is successfully created. * Current Limitation: At this stage (v3.4.2 through 4.0.x), only one automated cluster snapshot can be retained in the storage volume at any given time. 2. Retention for
information_schema.cluster_snapshot_jobs
The

cluster_snapshot_jobs

table stores the job execution history (the logs of successes and failures) rather than the snapshots themselves. * Does it store records continuously? No. To prevent the FE metadata from growing indefinitely, StarRocks automatically purges historical job records. * Retention Period: While there isn't a dedicated "snapshot-only" retention toggle, these records typically follow the global FE configuration for historical jobs: * history_job_keep_max_second
: (Default: 604,800 seconds / 7 days). Records older than this are purged. * label_keep_max_num
: (Default: 1000). If you have a high frequency of snapshots (default is every 10 mins, which is ~144/day), you will hit the 1000-record limit in about a week. How to change the snapshot frequency If you want to reduce the noise in your job history or save on storage operations, you can increase the interval between snapshots:

Copy code

sql
-- Change interval to 1 hour (3600 seconds)
ADMIN SET FRONTEND CONFIG ("automated_cluster_snapshot_interval_seconds" = "3600");

information_schema.cluster_snapshots

| 1 Job History/Logs |

information_schema.cluster_snapshot_jobs

| ~7 days or ~1000 records References • administration/cluster_snapshot.md • administration/management/FE_configuration.md References • administration/cluster_snapshot.md • administration/management/BE_configuration.md • administration/management/FE_configuration.md

Mohan

09/02/2026, 8:53 AM

@Rocky does the snapshot run only within FE node or does it affect any of the running things in compute nodes?

Rocky

09/02/2026, 8:53 AM

The Automated Cluster Snapshot in shared-data mode is designed to be non-disruptive and is primarily managed by the Frontend (FE) nodes. It does not negatively impact the performance of your compute nodes (CNs) or block your running queries. Here is the breakdown of how it affects each component: 1. Frontend (FE) Impact The FE is the primary "worker" for cluster snapshots. * Trigger: A snapshot is triggered every time the FE Leader completes a metadata checkpoint (by default every 10 minutes). * Workload: The FE serializes its current in-memory metadata (schemas, users, permissions, tablet locations) and uploads this "image" to your GCS bucket. * Performance: This is a background task. Since it uses the already-created checkpoint image, it doesn't "lock" the metadata or prevent other DDL/DML operations from continuing. 2. Compute Node (CN) Impact The CNs have no active role in the snapshot creation process. * No Data Copying: In a shared-data architecture, your actual data files are already residing in GCS. The "snapshot" doesn't involve copying or moving these massive data files; it simply records which versions of those files are current at that specific point in time. * No CPU/Memory Hit: Because the CNs aren't responsible for serializing metadata or uploading the snapshot files, you won't see a spike in CPU or memory on your compute layer during a snapshot. * No Query Blocking: Snapshots do not require global locks on tables. Queries continue to run against the data versions identified in the metadata, even while a new snapshot is being recorded. 3. Why it is "Invisible" The snapshot mechanism leverages StarRocks' MVCC (Multi-Version Concurrency Control): * In shared-data mode, all data files in GCS are immutable. _ When a snapshot is taken, the system essentially says: _"As of timestamp X, the valid data files for Table A are Version 1, 2, and 5."* * Because those files never change, the "snapshot" is essentially a small metadata reference pointing to existing files, making it extremely fast and lightweight. Summary Component | Role in Snapshot | Performance Impact FE (Leader) | Serializes metadata and uploads to GCS. | Low (background process). FE (Follower) | None. | Zero. CN (Compute) | None. | Zero. Running Queries | None. | Zero (no locks). Note on Restoration: While creating a snapshot is invisible, restoring from one is a heavy operation that requires stopping the cluster, clearing local caches/metadata, and restarting nodes from the snapshot state. References • administration/cluster_snapshot.md • quick_start/shared-data.md • administration/management/FE_configuration.md

Mohan

09/02/2026, 9:28 AM

@Rocky how to recover the cluster snapshot, my pvc deleted, already earlier snapshot pushed to bucket. how can I recover that?

Rocky

09/02/2026, 9:29 AM

If your FE PVC was deleted in a shared-data cluster, you can recover the entire cluster state (metadata and data pointers) using the Automated Cluster Snapshot stored in your bucket. Since the PVC is gone, the FE node is starting "fresh" and does not know which cluster it belongs to. You must provide this information via a configuration file during startup. Recovery Steps 1. Locate the Snapshot URI Go to your bucket and find the path to the snapshot. It typically looks like:

gs://<your-bucket>/<cluster-id>/meta/image/automated_cluster_snapshot_<timestamp>

(Or you can just point to the
meta
directory if you want the system to pick the latest). 2. Create the
cluster_snapshot.yaml
file In your FE configuration directory (

fe/conf/

), create a file named

cluster_snapshot.yaml

. This file tells the "blank" FE where to download the brain of the cluster from. Template for GCS:

Copy code

yaml
cluster_snapshot:
    # The URI found in Step 1
    cluster_snapshot_path: <gs://my-bucket/starrocks/meta>
    # Must match the name of the storage volume in your original cluster
    storage_volume_name: builtin_storage_volume 

# Define the connection to your bucket
storage_volumes:
  - name: builtin_storage_volume
    type: GCS
    location: <gs://my-bucket/starrocks/>
    properties:
      - key: gcs.service_account_email
        value: "<mailto:your-sa@project.iam.gserviceaccount.com|your-sa@project.iam.gserviceaccount.com>"
      - key: gcs.service_account_private_key_id
        value: "..."
      - key: gcs.service_account_private_key
        value: "..."

3. Start the Leader FE with the Recovery Flag You must start the first FE node (the new Leader) with a special flag that triggers the snapshot download:

Copy code

bash
./fe/bin/start_fe.sh --cluster_snapshot --daemon

* The FE will read

cluster_snapshot.yaml

, connect to the bucket, download the metadata image, and reconstruct its local

fe/meta

directory. * Once it is up, you can log in via MySQL and verify your databases and tables are visible. 4. Rejoin other FE and CN nodes Once the Leader is healthy: 1. Other FEs: Clear their

meta/

directories (if any exist) and start them using the

--helper

flag pointing to the restored Leader. 2. CNs: Clear their local

storage/

directories (to wipe old cache) and start them normally. They will automatically reconnect to the restored FE Leader. Important Notes * Storage Volume Name: The

storage_volume_name

in the YAML must match the name used in your original cluster (e.g.,

builtin_storage_volume

or whatever you named it during initial setup). * Cleaning Local Meta: If you tried to restart the FE before this and it created a fresh, empty

meta

folder, delete that folder entirely before running the command in Step 3. * Kubernetes Operator: If you are using the StarRocks Operator (v1.10.0+), you can perform this by setting the

disasterRecovery

field in your

StarRocksCluster

YAML rather than manually editing files. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • deployment/shared_data/gcs.md References • administration/cluster_snapshot.md • faq/shared_data_faq.md • deployment/shared_data/gcs.md • deployment/shared_data/minio.md • deployment/shared_data/s3.md

Mohan

09/02/2026, 9:42 AM

@Rocky my service is running in k8s, it is showing service already running root@starrocks-fe-0:/opt/starrocks# root@starrocks-fe-0:/opt/starrocks# ./fe/bin/start_fe.sh --cluster_snapshot --daemon Frontend running as process 98. Stop it first. root@starrocks-fe-0:/opt/starrocks# kill -9 98 root@starrocks-fe-0:/opt/starrocks# command terminated with exit code 137 how can I recover now?

Rocky

09/02/2026, 9:42 AM

In a Kubernetes environment, manual recovery using

kill -9

and running scripts inside the pod will not work because the Kubernetes Operator (or the StatefulSet controller) will immediately restart the pod as a "fresh" instance, seeing your manual process as a failure. To recover a shared-data cluster where the FE PVC is lost, you must use the Disaster Recovery feature of the StarRocks Operator. This tells the Operator to handle the shutdown and the

--cluster_snapshot

flag for you. Step 1: Prepare the
cluster_snapshot.yaml
You need to wrap the snapshot configuration into a Kubernetes manifest. If you are using the StarRocks Operator (v1.10.0+), you add this directly to your

StarRocksCluster

CRD. Step 2: Update your StarRocksCluster YAML Edit your

StarRocksCluster

resource (e.g.,

kubectl edit starrockscluster <your-cluster-name>

) and add the

disasterRecovery

section under

spec

. Example for GCS:

Copy code

yaml
spec:
  # ... other fields (starRocksFeSpec, etc.) ...
  disasterRecovery:
    enabled: true
    generation: 1  # Increment this number if you need to run recovery again
    yaml: |
      cluster_snapshot:
          cluster_snapshot_path: <gs://your-bucket/starrocks/meta>
          storage_volume_name: builtin_storage_volume 
      storage_volumes:
        - name: builtin_storage_volume
          type: GCS
          location: <gs://your-bucket/starrocks/>
          properties:
            - key: gcs.service_account_email
              value: "<mailto:your-sa@project.iam.gserviceaccount.com|your-sa@project.iam.gserviceaccount.com>"
            - key: gcs.service_account_private_key_id
              value: "..."
            - key: gcs.service_account_private_key
              value: "..."

Step 3: Apply the changes When you save this change (

kubectl apply

or save the editor), the Operator will: 1. Stop the existing FE pods. 2. Mount the provided configuration. 3. Restart the Leader FE pod with the

--cluster_snapshot

flag automatically. 4. Download the metadata from GCS and reconstruct the FE state. Step 4: Verify and Clean up 1. Check the FE logs to ensure recovery is progressing:

kubectl logs -f <fe-pod-name>

2. Once the cluster is back and healthy (you can log in via MySQL), set
enabled: false
in the

disasterRecovery

section of your YAML. If you leave it set to

true

, the Operator might attempt recovery again on the next pod restart. Why your manual attempt failed: * Self-healing: Kubernetes sees process 98 exit and thinks the pod crashed, so it restarts the container immediately. * Ephemeral Storage: Even if you succeeded in starting the FE manually, any metadata it downloaded would be lost the moment the pod restarts again because the PVC is missing or incorrect. Using the Operator ensures the metadata is restored to the volume correctly. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • deployment/shared_data/gcs.md References • administration/cluster_snapshot.md • administration/Meta_recovery.md • deployment/shared_data/gcs.md • faq/shared_data_faq.md

Mohan

09/02/2026, 10:04 AM

@Rocky I've deployed the cluster with helm, I can see thse in helm chart.

Copy code

# Note:
  #  1. If you are using an existing StarRocks cluster, you need to clean up the meta of the FE component and the data of the CN
  #     component before enabling disaster recovery. So it is better to use an empty StarRocks cluster to start disaster recovery.
  #  2. After disaster recovery, Operator will reboot the cluster as a normal cluster automatically, so if you need more checks by yourself,
  #     you can
  #       1. Set the replicas of FE component to 1
  #       2. Set enabledBe and enabledCn to be false to disable to deploy BE and CN components.
  #  3. Generation field is used to run multiple times for disaster recovery. For example, if the last disaster recovery is
  #     not what you want, you can modify related configurations and increase the generation value to run a new disaster recovery.
  disasterRecovery:
    # enabled: true
    # generation: 1
  # componentValues field is used to define values for all starrocks cluster components, including starrocksFESpec,
  # starrocksBeSpec, starrocksCnSpec, not including starrocksFeProxySpec. So that you do not need to modify them in
  # their own spec.
  # Note:
  #   1. The values in their own spec will take precedence over the values in this field.
  #   2. The values in their own spec will replace all the values in this field, not merge.
  componentValues:
    image:
      tag: "3.5-latest"
    # hostAliases allows adding entries to /etc/hosts inside the containers.
    hostAliases: []
      # - ip: "127.0.0.1"

how can I get the metadata now, I have the details of bucket where snapshot is written

Rocky

09/02/2026, 10:04 AM

To recover your cluster using the Helm chart, you need to populate the

disasterRecovery

section in your

values.yaml

file. This tells the StarRocks Operator to restart the FE with the necessary "recovery" instructions. 1. Locate your Snapshot Path Look into your GCS bucket. The automated snapshots are typically stored in a path like this:

gs://<your-bucket>/<cluster-id>/meta/image/automated_cluster_snapshot_<timestamp>

Tip: You can point to the

meta

directory (e.g., `gs://your-bucket/starrocks/meta`), and StarRocks will automatically find the latest snapshot within it. 2. Update your
values.yaml
Add the following configuration to your Helm

values.yaml

. Note that the

yaml

field is a multi-line string (indicated by the

symbol) that contains the actual recovery logic.

Copy code

yaml
starrocks:
  # ... your existing configurations ...

  disasterRecovery:
    enabled: true
    generation: 1 # Increment this every time you want to retry recovery
    yaml: |
      cluster_snapshot:
          # Point to the 'meta' folder in your bucket
          cluster_snapshot_path: <gs://your-bucket/path-to-your-data/meta>
          # Must match the name of the storage volume in your original cluster
          # (Default for shared-data is usually builtin_storage_volume)
          storage_volume_name: builtin_storage_volume 
      
      storage_volumes:
        - name: builtin_storage_volume
          type: GCS
          location: <gs://your-bucket/path-to-your-data/>
          properties:
            - key: gcs.service_account_email
              value: "<mailto:your-service-account@project.iam.gserviceaccount.com|your-service-account@project.iam.gserviceaccount.com>"
            - key: gcs.service_account_private_key_id
              value: "your-key-id"
            - key: gcs.service_account_private_key
              value: "-----BEGIN PRIVATE KEY-----\n...\n-----END PRIVATE KEY-----\n"

3. Apply the changes Run the helm upgrade command to apply the recovery configuration:

Copy code

bash
helm upgrade <release-name> starrocks/starrocks -f values.yaml

4. What happens next? 1. Shutdown: The Operator will gracefully stop the current FE and CN pods. 2. Recovery Mode: It will start the Leader FE with the

--cluster_snapshot

flag and the configuration you provided. 3. Metadata Download: The FE will download the metadata from GCS into its new PVC. 4. Automatic Restart: Once the metadata is restored, the Operator will automatically restart the cluster in "normal" mode. Important Success Tips: * Storage Volume Name: If you didn't specifically name your storage volume when you first set up the cluster, it is almost certainly named

builtin_storage_volume

. * Scale Down First (Optional but Recommended): As noted in your Helm chart comments, if the recovery is complex, you can temporarily set

starrocksFESpec.replicas: 1

and

enabledCn: false

in your

values.yaml

until the Leader FE is healthy. * Clean Up: Once you verify your data is back, remember to set

disasterRecovery.enabled: false

in your

values.yaml

so the cluster doesn't try to "recover" again on the next pod restart. References • administration/cluster_snapshot.md • deployment/sr_operator.md • administration/Meta_recovery.md • deployment/shared_data/gcs.md References • quick_start/helm.md • administration/cluster_snapshot.md • administration/stargo.md

Mohan

09/02/2026, 10:52 AM

@Rocky tatus: Disaster Recovery Status: Observed Generation: 1 Phase: todo Reason: cluster_snapshot.yaml is not mounted Start Timestamp: 1788346200 Phase: failed Reason: error from FE controller: cluster_snapshot.yaml is not mounted Star Rocks Cn Status: getting this error I've added in helm values this way

Copy code

.
  disasterRecovery:
    enabled: true
    generation: 1
    yaml: |
      cluster_snapshot:
        # Point to the 'meta' folder in your bucket
        cluster_snapshot_path: s3://<bucket>/d2b141e8-78d8-4279-b182-9c906378798d/meta
        # Must match the name of the storage volume in your original cluster
        # (Default for shared-data is usually builtin_storage_volume)
        storage_volume_name: gcs_snapshot_volume

        storage_volumes:
          - name: gcs_snapshot_volume
            type: S3
            location: s3://<bucket>
            properties:
              - key: aws.s3.region
                value: "asia-south1"

Rocky

09/02/2026, 10:52 AM

The error "cluster\_snapshot.yaml is not mounted" usually occurs because the StarRocks Operator failed to parse or apply the YAML configuration you provided in the

disasterRecovery.yaml

field. Based on the snippet you shared, there are three specific issues you need to fix: 1. Correct the Indentation (Crucial) In your YAML,

storage_volumes

is indented inside the

cluster_snapshot

block. They must be sibling keys (at the same level). 2. Fix the Bucket URI and Region * Characters: Your snippet shows

<bucket>

. You must use the actual bucket name without the

symbols or HTML entities. * Region: You are using

asia-south1

. This is a Google Cloud region. If you are using S3, the region should be something like

ap-south-1

. If you are using GCS, the

type

should be

GCS

. 3. Add Credentials The FE needs permission to read from the bucket. You must provide access keys or a service account. ────────── Corrected
values.yaml
snippet:

Copy code

yaml
  disasterRecovery:
    enabled: true
    generation: 2  # Increment this to 2 to trigger a retry
    yaml: |
      cluster_snapshot:
        # Point to the 'meta' folder. Remove the < > symbols.
        cluster_snapshot_path: <s3://your-bucket-name/d2b141e8-78d8-4279-b182-9c906378798d/meta>
        # Must match the name of the storage volume in your original cluster
        storage_volume_name: s3_snapshot_volume

      # NOTE: storage_volumes is at the same indentation level as cluster_snapshot
      storage_volumes:
        - name: s3_snapshot_volume
          type: S3
          location: <s3://your-bucket-name/>
          properties:
            - key: aws.s3.region
              value: "ap-south-1" # Use correct AWS region code
            - key: aws.s3.access_key
              value: "YOUR_ACCESS_KEY"
            - key: aws.s3.secret_key
              value: "YOUR_SECRET_KEY"

Steps to Recover: 1. Update the YAML: Fix the indentation and bucket names as shown above. 2. Increase Generation: Since the previous attempt failed, change

generation: 1

generation: 2

. This forces the Operator to re-process the recovery request. 3. Apply: Run

helm upgrade <release> starrocks/starrocks -f values.yaml

. 4. Monitor Logs: Check the FE pod logs:

kubectl logs -f <fe-pod-name>

. You should see it downloading the snapshot image. Important Note: The

storage_volume_name

must be exactly the same as the one used when the cluster was first created (default is usually

builtin_storage_volume

unless you changed it). References • administration/cluster_snapshot.md • deployment/shared_data/s3.md • deployment/sr_operator.md • administration/Meta_recovery.md • faq/shared_data_faq.md References • administration/cluster_snapshot.md • deployment/helm.md • deployment/sr_operator.md • quick_start/helm.md

Mohan

09/02/2026, 11:02 AM

@Rocky this is giving this the operator logs

Copy code

c/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2026-09-02T18:56:31.064+0800	ERROR	StarRocksClusterReconciler	sub controller reconciles spec failed	{"name": "starrocks", "namespace": "starrocks-shared", "subController": "feController", "error": "cluster_snapshot.yaml is not mounted"}
<http://github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile|github.com/StarRocks/starrocks-kubernetes-operator/pkg/controllers.(*StarRocksClusterReconciler).Reconcile>
	/go/src/app/pkg/controllers/starrockscluster_controller.go:94
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:122
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:323
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:274
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:235
2026-09-02T18:56:31.081+0800	ERROR	Reconciler error	{"controller": "starrockscluster", "controllerGroup": "<http://starrocks.com|starrocks.com>", "controllerKind": "StarRocksCluster", "StarRocksCluster": {"name":"starrocks","namespace":"starrocks-shared"}, "namespace": "starrocks-shared", "name": "starrocks", "reconcileID": "8d265f68-64de-4a3e-8740-5ef538c2a47a", "error": "cluster_snapshot.yaml is not mounted"}
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler>
	/go/src/app/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:329
<http://sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem|sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem>
	/go/src/app/vendor/sigs.k8s.io/cont

I don't see yml of cluster_snapshot in starrockscluster type in k8s even though I added in values.yml and applied through helm here is my cofig from values.yml

Copy code

disasterRecovery:
    enabled: true
    generation: 1
    yaml: |
      cluster_snapshot:
        # Point to the 'meta' folder in your bucket
        cluster_snapshot_path: s3://<bucket>/d2b141e8-78d8-4279-b182-9c906378798d/meta
        # Must match the name of the storage volume in your original cluster
        # (Default for shared-data is usually builtin_storage_volume)
        storage_volume_name: gcs_snapshot_volume

      storage_volumes:
        - name: gcs_snapshot_volume
          type: S3
          location: s3://<bucket>
          properties:
            - key: aws.s3.region
              value: "asia-south1"
            - key: aws.s3.endpoint
              value: "<https://storage.googleapis.com>"
            - key: aws.s3.access_key

Rocky

09/02/2026, 11:02 AM

The error "cluster_snapshot.yaml is not mounted" and the fact that you do not see the

yaml

field in your

StarRocksCluster

(via

kubectl get src -o yaml

) both point to the same issue: Your Helm chart version is likely too old. The automated Disaster Recovery feature, including the

yaml

configuration field, was introduced in StarRocks Operator v1.10.0. If you are using an older version of the Helm chart (e.g., 1.8.x or 1.9.x), the Helm template simply ignores the

yaml

property in your

values.yaml

and does not pass it to Kubernetes. To recover, follow these steps: 1. Upgrade your Helm Chart and Operator Check your current chart version using

helm list

. If it is below

1.10.0

, you must upgrade.

Copy code

bash
helm repo update starrocks
# Upgrade your release to the latest version (e.g., 1.11.x)
helm upgrade <release-name> starrocks/starrocks --version 1.11.7 -f values.yaml

Note: Ensure your Operator deployment is also updated to v1.10.0 or higher. 2. Fix the
values.yaml
content There are a few syntax issues in your snippet that will prevent StarRocks from parsing the recovery file: * Remove HTML Entities: Replace

<bucket>

with your actual bucket name (e.g.,

my-sr-bucket

). * Fix Region/Endpoint: You are using

asia-south1

(a GCS region) with

type: S3

. * If you are using AWS S3, use an AWS region (e.g.,

ap-south-1

). * If you are using GCS with an S3-compatible endpoint, ensure the credentials and endpoint are correct. * Indentation: Ensure

storage_volumes

is a sibling to

cluster_snapshot

(not nested inside it). Corrected
values.yaml
example:

Copy code

yaml
starrocks:
  disasterRecovery:
    enabled: true
    generation: 2  # Increment this to trigger a fresh attempt
    yaml: |
      cluster_snapshot:
        cluster_snapshot_path: <s3://my-sr-bucket/d2b141e8-78d8-4279-b182-9c906378798d/meta>
        storage_volume_name: s3_snapshot_volume

      storage_volumes:
        - name: s3_snapshot_volume
          type: S3
          location: <s3://my-sr-bucket/>
          properties:
            - key: aws.s3.region
              value: "ap-south-1" 
            - key: aws.s3.endpoint
              value: "<https://s3.ap-south-1.amazonaws.com>"
            - key: aws.s3.access_key
              value: "YOUR_ACCESS_KEY"
            - key: aws.s3.secret_key
              value: "YOUR_SECRET_KEY"

3. Verify the CRD After running

helm upgrade

, verify that the

yaml

field now exists in the cluster spec:

Copy code

bash
kubectl get starrockscluster starrocks -n starrocks-shared -o jsonpath='{.spec.disasterRecovery.yaml}'

If this output is still empty, your Helm chart templates are not up to date. 4. Monitor the FE Logs Once the

yaml

is correctly applied, the Operator will automatically create a ConfigMap and mount it to the FE. Check the FE pod logs:

Copy code

bash
kubectl logs -f starrocks-fe-0 -n starrocks-shared

You should see the FE starting with the

--cluster_snapshot

flag and successfully downloading the metadata from S3. References • administration/cluster_snapshot.md • deployment/sr_operator.md • administration/Meta_recovery.md • deployment/shared_data/s3.md References • administration/cluster_snapshot.md • deployment/helm.md • deployment/sr_operator.md • quick_start/helm.md References • quick_start/helm.md • administration/cluster_snapshot.md

Mohan

09/02/2026, 11:16 AM

@Rocky I'm using 1.11.7 operator chart version

Copy code

on-host-new:~$
mohan_krishna_nobroker_in@bastion-host-new:~$ kubectl get starrockscluster starrocks -n starrocks-shared
NAME        PHASE    FESTATUS   BESTATUS   CNSTATUS   FEPROXYSTATUS
starrocks   failed   running               running    running
mohan_krishna_nobroker_in@bastion-host-new:~$
mohan_krishna_nobroker_in@bastion-host-new:~$ kubectl get starrockscluster starrocks -n starrocks-shared -o jsonpath='{.spec.disasterRecovery.yaml}'
mohan_krishna_nobroker_in@bastion-host-new:~$

I don't see any disasterRecovery.yaml, I see the state failed with above shared logs

Rocky

09/02/2026, 11:17 AM

The reason your

kubectl get

command returns nothing for

.spec.disasterRecovery.yaml

is that Helm is not finding that field in your
values.yaml
, so it is not including it when it generates the

StarRocksCluster

resource. Based on the snippets you've shared, there are two likely causes: 1. Incorrect Parent Key or Indentation The comments you quoted (

"componentValues field is used to define..."

) come from the starrocksCluster
block in the official Helm chart. If you are using the

starrocks/kube-starrocks

chart (the most common one), your configuration must be nested under that key. If your
values.yaml
looks like this, it will fail:

Copy code

yaml
# WRONG: Indented under nothing or the wrong key
disasterRecovery:
  enabled: true
  yaml: |
    ...

It should look like this:

Copy code

yaml
starrocksCluster:  # <--- This header is required
  disasterRecovery:
    enabled: true
    generation: 2  # Increment this to trigger a fresh reconciliation
    yaml: |
      cluster_snapshot:
        cluster_snapshot_path: <s3://your-bucket-name/path-to-meta>
        storage_volume_name: s3_snapshot_volume

      storage_volumes:
        - name: s3_snapshot_volume
          type: S3
          location: <s3://your-bucket-name/>
          properties:
            - key: aws.s3.region
              value: "ap-south-1"  # Use correct S3 region
            - key: aws.s3.access_key
              value: "YOUR_ACCESS_KEY"
            - key: aws.s3.secret_key
              value: "YOUR_SECRET_KEY"

2. Check for HTML Entities In your snippet, the bucket path contains

<bucket>

. If your local

values.yaml

literally contains these HTML entities (e.g., from a copy-paste error), Helm or the YAML parser might treat the line as invalid or empty. Replace them with the actual bucket name (e.g., `s3://my-snapshot-bucket/...`). ────────── How to Debug and Fix 1. Check the full Spec: Run this command to see the entire resource definition. If you don't see the

yaml

field under

spec.disasterRecovery

, Helm didn't apply it.

Copy code

bash
    kubectl get starrockscluster starrocks -n starrocks-shared -o yaml

2. Verify your Helm Release: Confirm which chart you are actually using:

Copy code

bash
    helm list -n starrocks-shared

* If the chart is

kube-starrocks

, use the

starrocksCluster:

key in your values. * If you are using the subchart

starrocks

directly,

disasterRecovery

should be at the root (no indentation). 3. Fix the S3 configuration: Your snippet shows

type: S3

but a region of

asia-south1

(which is a Google Cloud region). * If using AWS S3, use a region like

ap-south-1

. * If using GCS with S3-compatibility, the type is still

S3

, but ensure the endpoint is correct and your credentials are for the S3-compatible interoperability keys in GCS. Recommended Next Step: Update your

values.yaml

to ensure

disasterRecovery

is correctly nested under

starrocksCluster:

, replace the placeholder bucket names with real ones, and run the

helm upgrade

again. Then, verify the CR again with

kubectl get starrockscluster starrocks -o yaml

. References • administration/cluster_snapshot.md • deployment/sr_operator.md • administration/Meta_recovery.md • deployment/shared_data/s3.md References • quick_start/helm.md • administration/cluster_snapshot.md • deployment/sr_operator.md References • quick_start/helm.md • administration/cluster_snapshot.md

Rocky

09/02/2026, 11:17 AM

• administration/management/BE_configuration.md

Mohan

09/02/2026, 11:23 AM

@Rocky I see it is correctly nest4ed in values.yml but it is not being visible in the actual deployed starrockscluster spec: disasterRecovery: enabled: true generation: 2 starRocksCnSpec: annotations: app.starrocks.io/cn-config-hash: 536e5858 prometheus.io/path: /metrics prometheus.io/port: "8040" prometheus.io/scrape: "true" yml is not mounted errror

Rocky

09/02/2026, 11:23 AM

The reason

disasterRecovery.yaml

is not appearing in your

StarRocksCluster

spec—and why you are getting the "not mounted" error—is that the
StarRocksCluster
CRD does not actually have a
yaml
field under
disasterRecovery
. The

disasterRecovery

block in the spec only supports two fields:

enabled

and

generation

. To provide the configuration for the recovery, you must manually create a Kubernetes ConfigMap and mount it to the FE pods. How to Fix This 1. Create a ConfigMap for the recovery config Create a file named

cluster_snapshot.yaml

locally with your configuration. Crucially, ensure

storage_volumes

is at the same level as

cluster_snapshot

Copy code

yaml
# Save this as cluster_snapshot.yaml
cluster_snapshot:
  cluster_snapshot_path: <s3://your-bucket/path/to/meta>
  storage_volume_name: my_s3_volume

storage_volumes:
  - name: my_s3_volume
    type: S3
    location: <s3://your-bucket/>
    properties:
      - key: aws.s3.region
        value: "your-region"
      - key: aws.s3.endpoint
        value: "<https://s3.your-region.amazonaws.com>"
      - key: aws.s3.access_key
        value: "YOUR_KEY"
      - key: aws.s3.secret_key
        value: "YOUR_SECRET"

Apply it to your namespace:

Copy code

bash
kubectl create configmap cluster-snapshot-config --from-file=cluster_snapshot.yaml=cluster_snapshot.yaml -n starrocks-shared

2. Update your
values.yaml
to mount this ConfigMap The FE controller expects the file at

/opt/starrocks/fe/conf/cluster_snapshot.yaml

. Use the

configMaps

array in

starRocksFESpec

to mount it.

Copy code

yaml
starrocksCluster:
  disasterRecovery:
    enabled: true
    generation: 2  # Increment to trigger a new attempt
  
  starRocksFESpec:
    configMaps:
      - name: cluster-snapshot-config
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml

3. Apply the changes Run your helm upgrade:

Copy code

bash
helm upgrade starrocks starrocks/kube-starrocks -f values.yaml -n starrocks-shared

Why this happened In the StarRocks Operator (v1.10+), the

disasterRecovery

feature tells the Operator to start the FE in a special recovery mode (using the

--cluster_snapshot

flag). However, the Operator does not "inject" the YAML directly from the CRD. Instead, it validates that the file

cluster_snapshot.yaml

exists at the standard config path in the FE container. By adding the manual mount, you satisfy this requirement. Verification: After applying, check the FE pod: 1. Run

kubectl get pod <fe-pod> -n starrocks-shared -o yaml

and verify the ConfigMap is mounted under

volumeMounts

. 2. Check the FE logs:

kubectl logs <fe-pod> -n starrocks-shared

. You should see the FE starting up and processing the snapshot metadata. References • administration/cluster_snapshot.md • deployment/sr_operator.md • administration/Meta_recovery.md • deployment/shared_data/s3.md References • administration/cluster_snapshot.md • deployment/sr_operator.md • administration/Meta_recovery.md • deployment/shared_data/s3.md References • administration/cluster_snapshot.md • deployment/shared_data/s3.md • administration/management/FE_configuration.md • deployment/shared_data/minio.md

Mohan

09/02/2026, 12:30 PM

@Rocky after recovery, I can see empty snapshots running

Copy code

select * from information_schema.cluster_snapshots;
Empty set (0.01 sec)

and also showing the automated snapshot turned on

Copy code

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME gcs_snapshot_volume;
ERROR 1064 (HY000): Getting analyzing error. Detail message: Automated snapshot has been turn on.
mysql>

how can I fix this?

Rocky

09/02/2026, 12:30 PM

If you see an empty set in

information_schema.cluster_snapshots

but get an error saying it is already enabled, this is typical behavior immediately following a restoration. Why this is happening 1. Metadata State: When you restore a cluster from a snapshot, the metadata you restored contains the setting "Automated Snapshots: ON" from your original cluster. This is why the

ADMIN SET

command fails—the flag is already set in the FE's memory. 2. Snapshot Registry: The

information_schema.cluster_snapshots

view only tracks snapshots managed by the current running instance. Even though you recovered from a snapshot in your bucket, the new cluster doesn't automatically "import" that history into its local view until it successfully completes its first new snapshot cycle. 3. Timing: By default, automated snapshots are triggered every 600 seconds (10 minutes). If you just finished recovery, the first cycle likely hasn't run yet. How to fix and verify 1. Check for active or failed jobs Instead of looking at the snapshots themselves, look at the background jobs to see if the first one is currently running or if it failed (e.g., due to permissions):

Copy code

sql
SELECT * FROM information_schema.cluster_snapshot_jobs;

If you see a job in the

ERROR

state, check the

error_message

column for the reason. 2. Reset the automated snapshot cycle If you want to "force" the cluster to re-verify the storage volume and restart the cycle, turn it OFF and then back ON:

Copy code

sql
-- This will clear the state and purge old pointers
ADMIN SET AUTOMATED CLUSTER SNAPSHOT OFF;

-- This will re-initialize the connection to your volume
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME gcs_snapshot_volume;

3. Decrease the interval (Optional, for testing) If you don't want to wait 10 minutes to see if it works, you can temporarily lower the interval:

Copy code

sql
-- Set interval to 2 minutes (120 seconds)
ADMIN SET FRONTEND CONFIG ("automated_cluster_snapshot_interval_seconds" = "120");

4. Check FE Leader logs If no jobs appear in

cluster_snapshot_jobs

after 10 minutes, check the Leader FE's

fe.log

for lines containing `ClusterSnapshot`:

Copy code

bash
# In the FE pod
grep -i "ClusterSnapshot" logs/fe.log

Summary * Empty set? Normal for the first ~10 minutes after recovery. * Error "Already turned on"? Normal because the restored metadata kept the old setting. * Solution: Turn it

OFF

and

ON

again to reset the task, or wait 10 minutes for the first job to finish. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md • administration/management/Backup_and_restore.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md • administration/Meta_recovery.md References • administration/cluster_snapshot.md • quick_start/helm.md • administration/management/Backup_and_restore.md

Mohan

09/02/2026, 7:33 PM

@Rocky even though I have latest cluster snapshot succeeded in bucket, on disaster recovery of incrementing generation, it is giving old databases/tables, doesn't have current day databases/tables even though few mins back there were cluster snapshot succeeded and I can see cluster snapshot path with this path - s3://<bucket>/d2b141e8-78d8-4279-b182-9c906378798d/meta am I missing something here? how can I get latest snapshot on recovery?

Rocky

09/02/2026, 7:33 PM

The reason you are seeing old data during recovery is likely because pointing to the base

/meta

directory relies on the FE's internal logic to find the latest image, which can sometimes be delayed by the metadata checkpoint cycle. To ensure you get the absolute latest state, follow these steps: 1. Identify the latest Snapshot URI Instead of pointing to the generic

/meta

folder, go to your S3 bucket and find the specific timestamped folder. * Path:

s3://<bucket>/d2b141e8-78d8-4279-b182-9c906378798d/meta/image/

* Inside this folder, look for the directory with the largest timestamp (e.g.,

automated_cluster_snapshot_1725283200000

). 2. Update
cluster_snapshot.yaml
to the specific path Modify your ConfigMap (or the

yaml

field if you found a way to make it work) to point directly to that timestamped folder. This removes any ambiguity.

Copy code

yaml
cluster_snapshot:
    # Point to the SPECIFIC timestamped folder
    cluster_snapshot_path: s3://<bucket>/d2b141e8-78d8-4279-b182-9c906378798d/meta/image/automated_cluster_snapshot_1725283200000
    storage_volume_name: gcs_snapshot_volume

3. Ensure a Clean Metadata Slate When you increment the

generation

in the

StarRocksCluster

spec, the Operator restarts the FE with the

--cluster_snapshot

flag. However, if the old metadata still exists on the Persistent Volume (PV), the FE might occasionally attempt to recover from local journals first. To be safe: 1. Scale down the FE to 0 temporarily via Helm or

kubectl scale

. 2. Delete the data in the FE metadata volume (if possible) or ensure the Operator-triggered restart is definitely using the new config. 3. Update the ConfigMap with the new specific path. 4. Scale the FE back up (or just increment

generation

if you are keeping the cluster running). Why "Latest" might be missing If the data from "minutes ago" is missing, check these two factors: * Checkpoint Interval: StarRocks creates a snapshot only after a metadata checkpoint (when a new

image.xxxx

file is created). By default, this happens every 50,000 journal entries or at a specific time interval. If your "latest" data hasn't been "checkpointed" into an image yet, it won't be in the snapshot. * Snapshot Interval: The FE configuration

automated_cluster_snapshot_interval_seconds

(default 600s / 10m) controls how often the snapshot is pushed to S3. Even if a checkpoint happens, the snapshot won't move to S3 until this timer hits. To verify what is actually being loaded: Check the Leader FE logs during startup:

Copy code

bash
kubectl logs <fe-pod-name> | grep "recover from cluster snapshot"

It will log the exact URI and the

journal_id

it is restoring from. Compare this

journal_id

with your original cluster's

SHOW FRONTENDS

to see how far behind it is. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • administration/management/FE_configuration.md • administration/management/Backup_and_restore.md References • administration/cluster_snapshot.md • administration/Meta_recovery.md • administration/management/Backup_and_restore.md • deployment/shared_data/s3.md References • administration/cluster_snapshot.md • administration/Meta_recovery.md

2 Views

Open in Slack

Previous Next