@Rocky whats default cluster snapshot interval? and can I modify that time?

The default interval for automated cluster snapshots is 600 seconds (10 minutes). You can modify this interval by changing the FE configuration parameter

automated_cluster_snapshot_interval_seconds

. Since this parameter is mutable, you can update it dynamically using the following SQL command:

sql
ADMIN SET FRONTEND CONFIG ("automated_cluster_snapshot_interval_seconds" = "YourValue");

Key details to keep in mind: * Availability: This feature was introduced in v3.4.2 and is only available for shared-data clusters. * Activation: Automated snapshots are disabled by default. You must enable them first using:

sql
    ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON;

* Retention: Currently, only one automated cluster snapshot is retained at a time. When a new snapshot is created, the previous one is automatically purged. References • administration/management/FE_configuration.md • administration/cluster_snapshot.md • administration/management/BE_configuration.md

@Rocky how to check if automated cluster snap is on or off?

There is no single "status" variable (like a boolean) in the standard FE configuration that explicitly states "ON" or "OFF." Instead, you can verify if automated cluster snapshots are active by checking the system views for activity and snapshots: 1. Check for Active Snapshots Query the

information_schema.cluster_snapshots

view. If the feature is enabled and has completed at least one cycle, you will see an entry with the type

automated

.

sql
SELECT * FROM information_schema.cluster_snapshots 
WHERE snapshot_type = 'automated';

* If a row exists: The feature is ON and has successfully created a snapshot. * If no row exists: The feature is either OFF, or it is ON but hasn't reached its first interval yet (default 10 minutes). 2. Check Snapshot Job History To see the actual execution status and history (even if a snapshot failed to upload), query the

cluster_snapshot_jobs

view:

sql
SELECT * FROM information_schema.cluster_snapshot_jobs 
ORDER BY created_time DESC;

Look for jobs where the

snapshot_name

starts with

automated_cluster_snapshot_

. If you see recurring jobs with a

FINISHED

state, the feature is active. 3. Verification by "Re-enabling" Because

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON

is idempotent, you can run it again to ensure it is active. If you need to specify or verify the storage volume being used, you can include it:

sql
ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON STORAGE VOLUME your_volume_name;

Summary of Indicators * Feature Status: Managed via

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON/OFF

. * Validation: Check

information_schema.cluster_snapshots

. * Troubleshooting: Check

information_schema.cluster_snapshot_jobs

for error messages (e.g., upload failures to S3/Azure). References • administration/cluster_snapshot.md • administration/management/FE_configuration.md References • administration/management/FE_configuration.md • administration/cluster_snapshot.md

@Rocky I enabled cluster snapshot but dont see any records in information_schema.cluster_snapshots

If you enabled the feature but don't see any records in

information_schema.cluster_snapshots

, it is likely due to one of the following reasons: 1. Check for Errors in Job History The

cluster_snapshots

view only shows successful and active snapshots. If the snapshot process failed (e.g., due to storage permission issues), you need to check the job history view instead:

sql
SELECT * FROM information_schema.cluster_snapshot_jobs 
ORDER BY created_time DESC;

Look at the

state

and

error_message

columns. This is where you will find details if the system tried to create a snapshot but encountered an error reaching your S3/Object Storage. 2. Dependency on Metadata Checkpoints Automated snapshots are not triggered purely by a clock. According to the documentation, a snapshot is created after a metadata checkpoint is completed. * If your cluster has very little activity, a checkpoint might not have been triggered yet. * The default interval is 10 minutes, but the snapshot only happens once the FE successfully writes a new metadata image. 3. Verify Shared-Data Architecture This feature is only supported on shared-data clusters (Cloud-Native). If you are running a standard shared-nothing cluster (where data is stored on local FE/BE disks), automated cluster snapshots will not function. 4. Storage Volume Configuration If you didn't specify a storage volume when enabling the feature, StarRocks uses the default storage volume. * Ensure a default storage volume is set:

SHOW STORAGE VOLUMES;

* Ensure the FE has the correct permissions (IAM roles or Secret Keys) to write to that volume. Summary Checklist: 1. Wait: Ensure at least 10–20 minutes have passed. 2. Jobs: Run

SELECT * FROM information_schema.cluster_snapshot_jobs;

to see if a job was created and failed. 3. Logs: Check the

fe.log

on the leader FE for keywords like

ClusterSnapshot

or

Checkpoint

to see if any background errors are occurring. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md References • administration/cluster_snapshot.md • administration/management/FE_configuration.md • administration/management/Backup_and_restore.md

@Rocky upload image failed, err msg: Failed to copy local /opt/starrocks/fe/meta/image to s3://dev-starrocks-fe/snapshot/fe919e73-9b67-4920-b15b-fad76a32c664/meta/image/automated_cluster_snapshot_1781595384808

The error "Failed to copy local /opt/starrocks/fe/meta/image to s3://..." indicates that the FE leader failed to upload the metadata snapshot to your S3 storage volume. Based on recent community reports and known issues (specifically in StarRocks v3.4.x), here are the most likely causes and solutions: 1. Known Bug: Path-Style Access (MinIO/S3-Compatible) If you are using MinIO or another S3-compatible storage that requires path-style access (e.g.,

endpoint/bucket

instead of

bucket.endpoint

), there is a known issue where the FE does not correctly inherit the

aws.s3.enable_path_style_access

property from the Storage Volume configuration. * Symptom: Check your

fe.log

for a

java.net.UnknownHostException

. If you see a domain like

<http://bucketname.your-minio-endpoint.com|bucketname.your-minio-endpoint.com>

failing to resolve, this is the issue. * Workaround: You must explicitly enable path-style access for the FE by adding the following to

fe/conf/core-site.xml

on all FE nodes and restarting them:

xml
    <property>
      <name>fs.s3a.path.style.access</name>
      <value>true</value>
    </property>

2. Missing S3 Write Permissions The FE leader node performs the upload. Ensure the credentials (AK/SK or IAM Instance Profile) associated with your storage volume have the following permissions on the bucket `dev-starrocks-fe`: *

s3:PutObject

*

s3:GetAttributes

*

s3:ListBucket

3. Storage Volume Verification Verify that the storage volume is correctly configured. Run:

sql
DESC STORAGE VOLUME your_volume_name;

Check that the

aws.s3.endpoint

and

aws.s3.region

are correct. If the endpoint is internal (e.g.,

<http://s3.dualstack.us-east-1.amazonaws.com|s3.dualstack.us-east-1.amazonaws.com>

), ensure the FE leader has network route access to it. 4. Check FE Logs for Detail The error message in

information_schema.cluster_snapshot_jobs

is often a summary. For the full stack trace, check the

fe.log

on your Leader FE:

bash
grep -A 20 "Failed to copy local" fe/log/fe.log

This will tell you if the failure is a

403 Forbidden

(permission),

404 Not Found

(wrong bucket/path), or

UnknownHostException

(DNS/Path-style issue). Summary Checklist * Is it MinIO/Ceph? Apply the

core-site.xml

fix. * Is it AWS S3? Check if the FE node's IAM role or AK/SK has

PutObject

rights. * Is it v3.4.2 - v3.4.5? These versions are most susceptible to the path-style configuration bug mentioned in Step 1. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md References • administration/cluster_snapshot.md • administration/management/FE_configuration.md • administration/Meta_recovery.md

@Rocky can I run cluster snapshot manually to see whats the exact problem?

There is no direct command like

CREATE CLUSTER SNAPSHOT

in the current version of StarRocks. The "Cluster Snapshot" feature is designed as an automated background process that follows the FE metadata checkpoint cycle. However, you can manually trigger the snapshot logic by forcing a metadata checkpoint. This allows you to verify if your storage settings are working without waiting for the 10-minute (default) automated interval. 1. Manually Trigger the Snapshot Run the following command on your StarRocks cluster:

sql
ALTER SYSTEM CREATE IMAGE;

Why this works: The StarRocks documentation states that "each time FE creates a new metadata image after completing a metadata checkpoint, it automatically creates a snapshot." By running

CREATE IMAGE

, you force the FE to write its memory state to disk, which immediately triggers the cluster snapshot job (the upload to S3). 2. Monitor the Result After running the command, check the job history to see if the upload succeeded or failed with the same error:

sql
SELECT * FROM information_schema.cluster_snapshot_jobs 
ORDER BY created_time DESC;

3. Debugging the "Failed to copy" error If the job still fails with the "Failed to copy local... to s3" error after the manual trigger, focus on these three areas: * FE S3 SDK Compatibility: If you are using MinIO or a non-AWS S3 provider, the FE might be failing to use "Path-Style Access." Add

<property><name>fs.s3a.path.style.access</name><value>true</value></property>

to your

fe/conf/core-site.xml

and restart the FE. * Write Permissions: Ensure the IAM Role or Access Key used for the Storage Volume has

s3:PutObject

and

s3:GetAttributes

permissions on the

dev-starrocks-fe

bucket. * FE Leader Logs: For a more detailed stack trace, log into your Leader FE node and run:

bash
    grep -C 5 "ClusterSnapshot" /path/to/fe/log/fe.log

References • administration/cluster_snapshot.md • sql-reference/sql-statements/cluster-management/nodes_processes/ALTER_SYSTEM.md • administration/management/FE_configuration.md References • administration/cluster_snapshot.md • sql-reference/sql-statements/backup_restore/BACKUP.md