@Rocky Using

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON

will fail if run multiple times, as it is not idempotent. What can I do if I have an kubernetes deployment that I want to be able to bring up and down, where I have an init job that runs SQL to create my database and configure a few properties including turning on automated cluster snapshots? It is not realistic to have to manually go into the environment each time it is deployed and turn on snapshots, but I also can't leave the line in the init job, because if the init job is ever updated and rerun, it will fail to complete correctly.

Since StarRocks does not currently support an

IF NOT EXISTS

clause for the

ADMIN SET AUTOMATED CLUSTER SNAPSHOT

command, and it is a persistent setting that survives restarts (stored in the FE metadata), it will indeed throw an error if executed while already active. To make your Kubernetes init job idempotent and prevent it from failing on subsequent runs, you can use one of the following strategies: 1. Use the

--force

flag with the MySQL Client The simplest way to handle this in a Kubernetes init container script is to use the

-f

(or

--force

) flag in your

mysql

command. This tells the client to continue processing the rest of your SQL script even if a particular command fails.

bash
# Example in your init job entrypoint
mysql -h $FE_HOST -P $FE_PORT -u root -f < /scripts/init.sql

2. Wrap the command in a Bash condition If your init job runs a shell script that calls

mysql

, you can wrap the snapshot command in a logic block that ignores that specific error.

bash
# Attempt to enable, but ignore the error if it fails (already enabled)
mysql -h $FE_HOST -u root -e "ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON;" || true

If you want to be more precise and only ignore the "already enabled" error while failing on others:

bash
RESULT=$(mysql -h $FE_HOST -u root -e "ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON;" 2>&1)
if [[ $? -ne 0 && ! "$RESULT" =~ "already enabled" ]]; then
    echo "Snapshot configuration failed: $RESULT"
    exit 1
fi

3. Use a "Marker Table" for Idempotency If you want to handle this entirely within your SQL script, you can create a tracking table to ensure "run-once" logic for non-idempotent admin commands.

sql
CREATE DATABASE IF NOT EXISTS system_init_db;
CREATE TABLE IF NOT EXISTS system_init_db.setup_status (
    feature_name VARCHAR(64) PRIMARY KEY,
    is_enabled BOOLEAN
) ENGINE=OLAP 
PRIMARY KEY(feature_name) 
DISTRIBUTED BY HASH(feature_name);

-- Since StarRocks SQL doesn't support IF/THEN scripting blocks, 
-- you would typically use a separate small script to check this table:
-- SELECT count(*) FROM system_init_db.setup_status WHERE feature_name='automated_snapshots';

4. Check for existing snapshots (Detection) You can query

information_schema.cluster_snapshots

to see if snapshots are already being generated. Note that on a brand new cluster, this will be empty for the first few minutes until the first snapshot is taken (controlled by

automated_cluster_snapshot_interval_seconds

).

sql
SELECT count(*) FROM information_schema.cluster_snapshots;

5. Persistent Configuration (StarRocks v3.5+) If you are using the StarRocks Kubernetes Operator and a recent version (v3.5+), there is a

disasterRecovery

field in the

StarRocksCluster

CRD spec designed to manage these states. However, for manual SQL configuration, the shell-based wrapper (

|| true

) remains the most reliable and realistic way to ensure your init job completes successfully every time. Recommendation: Use the

mysql -f

flag or the

|| true

bash wrapper. Since the setting is stored in the FE metadata, once it is successfully set the first time, your cluster will continue taking snapshots even if the init job is skipped or fails on that line later. References • administration/management/FE_configuration.md • administration/cluster_snapshot.md • administration/management/BE_configuration.md

@Rocky Expand on the

disasterRecovery

field and how that can be used to set the automated snapshots.

Starting with StarRocks v3.5 and Operator v1.10.0, the

disasterRecovery

field was introduced to the

StarRocksCluster

Custom Resource Definition (CRD) to handle complex recovery workflows. However, there is a distinction between Enabling snapshots (recurring backups) and Disaster Recovery mode (performing a restore). 1. The

disasterRecovery

field in CRD In the StarRocks Operator, the

disasterRecovery

field is used to tell the Operator that the cluster is in a "recovery" state. When you set this, the Operator changes its reconciliation logic to allow a new cluster to bootstrap from existing metadata/data in object storage. Example Spec:

yaml
spec:
  disasterRecovery:
    enabled: true
    generation: 1

* What it does: It signals the Operator to start the FE in recovery mode (using the

--cluster_snapshot

flag internally) and prevents the Operator from overwriting or "fixing" the cluster state until the recovery process moves from

todo

to

done

. * What it does NOT do: It does not automatically run the SQL command

ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON

. You still need to enable the snapshot feature once via SQL. 2. Solving the Idempotency Issue in Init Jobs Since there is no "IF NOT EXISTS" for the snapshot command and it is not a simple FE configuration file setting (it's stored in FE metadata), you must handle the error in your init script. The most realistic way to handle this in a Kubernetes init job is to ignore the error specifically for that line. Method A: Use the

--force

flag (Recommended) If your init job runs a

.sql

file via the MySQL client, use the

-f

flag. This ensures that if the snapshot command fails (because it's already on), the script continues to the next command.

bash
mysql -h $FE_HOST -P 9030 -u root -f < /scripts/init.sql

Method B: Bash Wrapper with Error Handling If you want your script to be clean and only ignore the "already enabled" error:

bash
# Attempt to enable, but treat "already enabled" as success
RESULT=$(mysql -h $FE_HOST -u root -e "ADMIN SET AUTOMATED CLUSTER SNAPSHOT ON;" 2>&1)
if [[ $? -ne 0 && ! "$RESULT" =~ "already enabled" ]]; then
    echo "Failed to enable snapshots: $RESULT"
    exit 1
fi

3. Check FE Configurations You can also set the snapshot interval via FE configuration in your

StarRocksCluster

YAML, which is naturally idempotent. While this doesn't "turn on" the switch, it configures the behavior once it is on.

yaml
spec:
  starRocksFeSpec:
    configMapInfo:
      configMapName: fe-config
      key: fe.conf
# In your ConfigMap, you can set:
# automated_cluster_snapshot_interval_seconds = 600

Summary for Kubernetes Users 1. To Enable: Use a SQL script in your init job but wrap it in a

|| true

or use

mysql -f

. This is standard practice because StarRocks treats "enabling an already enabled feature" as an error. 2. To Restore: Use the

disasterRecovery

field in the Operator CRD. This tells the Operator to help the cluster recover from those snapshots if the environment is ever completely wiped. re-deployed from those snapshots if the environment is completely wiped and brought back up. wiped and brought back up. References • administration/cluster_snapshot.md • administration/management/FE_configuration.md