http://starrocks.io logo
Join Slack
Powered by
Hi SR team, I am following this doc <https://www.s...
# questions-and-troubleshooting
r
Hi SR team, I am following this doc https://www.starrocks.io/blog/starrocks-disaster-recovery-on-kubernetes-a-practical-guide-to-cluster-snapshot to test disaster recovery on Kubernetes. I am able to take the cluster snapshot and automated_cluster_snapshot_1771523838532 also created in s3 bucket. But when I am trying to restore it in the new cluster, the pod FE is not coming up. I am using starrocks cluster helm chart and this is my values.yaml:
Copy code
starrocksCluster:
  disasterRecovery:
    enabled: true
    generation: 1
starrocksFESpec:
  configMaps:
    - name: cluster-snapshot
      mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
      subPath: cluster_snapshot.yaml
configMaps:
  - name: cluster-snapshot
    data:
      cluster_snapshot.yaml: |
        cluster_snapshot:
          cluster_snapshot_path: <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532>
          storage_volume_name: builtin_storage_volume
        frontends: []
        compute_nodes: []
        storage_volumes:
          - name: builtin_storage_volume
            type: S3
            location: <s3://kubvir-asd21-starrocks-data>
            comment: my s3 volume
            properties:
              - key: aws.s3.region
                value: us-east-1
              - key: aws.s3.endpoint
                value: <https://s3.amazonaws.com>
              - key: aws_s3_use_instance_profile
                value: false
              - key: aws_s3_use_aws_sdk_default_behavior
                value: true
Not seeing anything useful in operator logs. FE pods are just giving these logs:
Copy code
[Fri Feb 20 02:40:32 CST 2026] Process conf file fe.conf ...
[Fri Feb 20 02:40:32 CST 2026] first start fe with meta not exist.
[Fri Feb 20 02:40:32 CST 2026] FE service kube-starrocks-fe-service.asdfg:9030 is not alive yet!
[Fri Feb 20 02:40:34 CST 2026] FE service kube-starrocks-fe-service.asdfg:9030 is not alive yet!
[Fri Feb 20 02:41:02 CST 2026] Timed out, no members detected ever, assume myself is the first node ..
[Fri Feb 20 02:41:02 CST 2026] first start with no meta run start_fe.sh with additional options: ' --host_type FQDN'
Also the status of recovery remains in in progress
Copy code
kubectl get src kube-starrocks -n asd21 -o yaml

-----
status:
  disasterRecoveryStatus:
    observedGeneration: 1
    phase: doing
    reason: disaster recovery is in progress
    startTimestamp: 1771527180
  phase: reconciling
  starRocksCnStatus:
    horizontalScaler: {}
    phase: reconciling
  starRocksFeProxyStatus:
    phase: reconciling
  starRocksFeStatus:
    creatingInstances:
    - kube-starrocks-fe-0
    phase: reconciling
    resourceNames:
    - kube-starrocks-fe
    serviceName: kube-starrocks-fe-service
Please guide.
t
@Rocky - what should be check when the FE is never coming online in a DR k8s cluster? These are the instructions used: https://www.starrocks.io/blog/starrocks-disaster-recovery-on-kubernetes-a-practical-guide-to-cluster-snapshot
Copy code
[Fri Feb 20 02:40:32 CST 2026] Process conf file fe.conf ...
[Fri Feb 20 02:40:32 CST 2026] first start fe with meta not exist.
[Fri Feb 20 02:40:32 CST 2026] FE service kube-starrocks-fe-service.asdfg:9030 is not alive yet!
[Fri Feb 20 02:40:34 CST 2026] FE service kube-starrocks-fe-service.asdfg:9030 is not alive yet!
[Fri Feb 20 02:41:02 CST 2026] Timed out, no members detected ever, assume myself is the first node ..
[Fri Feb 20 02:41:02 CST 2026] first start with no meta run start_fe.sh with additional options: ' --host_type FQDN'
Also the status of recovery remains in in progress
Copy code
kubectl get src kube-starrocks -n asd21 -o yaml

-----
status:
  disasterRecoveryStatus:
    observedGeneration: 1
    phase: doing
    reason: disaster recovery is in progress
    startTimestamp: 1771527180
  phase: reconciling
  starRocksCnStatus:
    horizontalScaler: {}
    phase: reconciling
  starRocksFeProxyStatus:
    phase: reconciling
  starRocksFeStatus:
    creatingInstances:
    - kube-starrocks-fe-0
    phase: reconciling
    resourceNames:
    - kube-starrocks-fe
    serviceName: kube-starrocks-fe-service
r
The issues you are encountering—FE logs showing "meta not exist" and the 
StarRocksCluster
status stuck in the 
doing
phase—typically indicate a disconnect between the Kubernetes volume restoration and the StarRocks process startup. Based on the blog post and the logs provided, here are the key areas to check: 1. Verify Metadata Restoration (The "Smoking Gun") The log 
first start fe with meta not exist
is the most critical clue. In a Disaster Recovery (DR) scenario, the FE pod should find the metadata restored from your snapshot. If it doesn't, it assumes it's a brand-new cluster and attempts to initialize a fresh state, which conflicts with a DR restore. * Check the PVC content: Exec into the FE pod (if it's running but failing) or a temporary "debug" pod and check if the metadata files exist in the expected directory (usually 
/opt/starrocks/fe/meta
).
Copy code
bash
    kubectl exec -it kube-starrocks-fe-0 -n asd21 -- ls -R /opt/starrocks/fe/meta
If this directory is empty or missing the 
image/
and 
bdb/
folders, your Velero restore or VolumeSnapshot did not correctly populate the PVC. * Check Mount Paths: Ensure that the 
meta_dir
defined in your 
fe.conf
(via ConfigMap) matches the volume mount path in the 
StarRocksCluster
spec. If they mismatch, the FE process looks in one place while the restored data is in another. 2. "FE service ... is not alive yet!" Timeout The message 
FE service kube-starrocks-fe-service.asdfg:9030 is not alive yet!
is part of the StarRocks Operator's entrypoint script. It tries to detect if other FE members are already up to determine if it should "join" or "init". * In a DR scenario where you are restoring the first FE, this timeout is expected. However, because it also doesn't find the metadata, it tries to start as a completely new leader of a new cluster. * If the FQDN or IP of the service has changed (e.g., you restored to a new namespace/cluster), the metadata from the old cluster might prevent the FE from starting unless it is forced. 3. StarRocks Operator 
disasterRecoveryStatus
 The phase 
doing
means the Operator has initiated the DR reconciliation logic but is waiting for the FE pods to reach a 
Ready
state. Since the FE is stuck in a loop or starting with "no meta," it never becomes 
Ready
, and the Operator remains in 
doing
. 4. Recovery Configuration Requirements Depending on how the snapshot was taken and the StarRocks version: * FQDN Changes: If the cluster FQDNs have changed (common when moving namespaces), the FE might need the 
bdbje_reset_election_group = true
parameter in the 
fe.conf
to allow the restored metadata to accept the new environment. * Cluster Snapshot Flag: If you are using the native StarRocks Cluster Snapshot feature (mentioned in the blog), the FE process must be started with the 
--cluster_snapshot
flag. Check if your 
StarRocksCluster
spec has the appropriate DR configuration to pass this flag. Recommended Troubleshooting Steps: 1. Check Velero/Snapshot Logs: Verify that the restore of the 
kube-starrocks-fe-meta-kube-starrocks-fe-0
PVC was successful. 2. Inspect Operator Logs: Check the logs of the 
starrocks-controller
pod. It will provide more detail on why the DR phase is stuck.
Copy code
bash
    kubectl logs -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator> -n <operator-namespace>
3. Manual Metadata Check: If the metadata is indeed present but the script says "not exist," verify the 
meta_dir
path in your 
fe.conf
. 4. Wait for Service Discovery: Ensure the K8s service 
kube-starrocks-fe-service
and its DNS are resolvable within the pod. If DNS is slow to propagate in the new cluster, the FE might time out prematurely. Note: If you are restoring a shared-data cluster (Compute-Storage separation), ensure the 
storage_volume
configuration in the restored cluster matches the original one so the FE can find the data in object storage. References
r
@Tony Wasson This is not helping. I also queried Rocky but I am doing the same thing suggested by the output of rocky agent but still not able to resolve the issue.
t
@Rumit Jain - I would investigate your meta data backup first. Make sure you have something in the meta - do you have any tables, catalogs or views configured? Critical log event clue: 
first start fe with meta not exist
r
@Tony Wasson @Rocky I am seeing these errors in the fe pod logs :
Copy code
2026-02-24T17:18:20.499721223Z 2026-02-25 01:18:20.494+08:00 INFO (main|1) [RestoreClusterSnapshotMgr.init():66] FE start to restore from a cluster snapshot (RESTORE_CLUSTER_SNAPSHOT=true)
2026-02-24T17:18:21.491220214Z 2026-02-25 01:18:21.490+08:00 INFO (main|1) [RestoreClusterSnapshotMgr.downloadSnapshot():153] Download cluster snapshot <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532> to local dir /opt/starrocks/fe/meta/image
2026-02-24T17:18:21.687999054Z 2026-02-25 01:18:21.687+08:00 INFO (main|1) [HdfsFsManager.getFileSystemByCloudConfiguration():676] could not find file system for path <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532> create a new one
2026-02-24T17:18:22.179915193Z 2026-02-25 01:18:22.179+08:00 INFO (main|1) [FileSystem.createFileSystemInternal():3731] [hadoop-ext] FileSystem.createFileSystem
2026-02-24T17:18:22.884533928Z 2026-02-25 01:18:22.882+08:00 WARN (main|1) [NativeCodeLoader.<clinit>():60] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2026-02-24T17:18:24.535324031Z 2026-02-25 01:18:24.534+08:00 WARN (main|1) [MetricsConfig.loadFirst():138] Cannot locate configuration: tried hadoop-metrics2-s3a-file-system.properties,hadoop-metrics2.properties
2026-02-24T17:18:24.586607023Z 2026-02-25 01:18:24.586+08:00 INFO (main|1) [MetricsSystemImpl.startTimer():378] Scheduled Metric snapshot period at 10 second(s).
2026-02-24T17:18:24.586842230Z 2026-02-25 01:18:24.586+08:00 INFO (main|1) [MetricsSystemImpl.start():191] s3a-file-system metrics system started
2026-02-24T17:18:27.905634618Z 2026-02-25 01:18:27.904+08:00 INFO (main|1) [FileSystem.createFileSystemInternal():3731] [hadoop-ext] FileSystem.createFileSystem
2026-02-24T17:18:28.875046297Z log4j:WARN No appenders could be found for logger (software.amazon.awssdk.thirdparty.org.apache.http.client.protocol.RequestAddCookies).
2026-02-24T17:18:28.875097989Z log4j:WARN Please initialize the log4j system properly.
2026-02-24T17:18:28.875105829Z log4j:WARN See <http://logging.apache.org/log4j/1.2/faq.html#noconfig> for more info.
2026-02-24T17:18:29.792488320Z 2026-02-25 01:18:29.784+08:00 ERROR (main|1) [HdfsFsManager.copyToLocal():1212] Exception while copy <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532> to local /opt/starrocks/fe/meta/image 
2026-02-24T17:18:29.792553812Z java.nio.file.AccessDeniedException: <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532>: getFileStatus on <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532>: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden (Service: S3, Status Code: 403, Request ID: KHTTQA2P316NY0PG, Extended Request ID: 3hfcDh6I/GZLR59RTz2wl9zQl1nNjtZhypVA5iARUxH5NrTkH+/0c7x5ojXFvHDENlSEbgJLcKk=):null
2026-02-24T17:18:29.792567003Z 	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:267) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792573023Z 	at org.apache.hadoop.fs.s3a.S3AUtils.translateException(S3AUtils.java:156) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792578993Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4101) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792585183Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:4007) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792591674Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getFileStatus$22(S3AFileSystem.java:3984) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792597524Z 	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.invokeTrackingDuration(IOStatisticsBinding.java:547) ~[hadoop-common-3.4.1.jar:?]
2026-02-24T17:18:29.792604494Z 	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:528) ~[hadoop-common-3.4.1.jar:?]
2026-02-24T17:18:29.792611024Z 	at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.trackDuration(IOStatisticsBinding.java:449) ~[hadoop-common-3.4.1.jar:?]
2026-02-24T17:18:29.792615994Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2865) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792620844Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.trackDurationAndSpan(S3AFileSystem.java:2884) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792628275Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus(S3AFileSystem.java:3982) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792635125Z 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:431) ~[hadoop-common-3.4.1.jar:?]
2026-02-24T17:18:29.792640315Z 	at org.apache.hadoop.fs.FileUtil.copy(FileUtil.java:370) ~[hadoop-common-3.4.1.jar:?]
2026-02-24T17:18:29.792646005Z 	at org.apache.hadoop.fs.FileSystem.copyToLocalFile(FileSystem.java:2755) ~[starrocks-hadoop-ext.jar:?]
2026-02-24T17:18:29.792671976Z 	at com.starrocks.fs.hdfs.HdfsFsManager.copyToLocal(HdfsFsManager.java:1205) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792678576Z 	at com.starrocks.fs.hdfs.HdfsService.copyToLocal(HdfsService.java:58) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792683856Z 	at com.starrocks.fs.HdfsUtil.copyToLocal(HdfsUtil.java:71) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792689017Z 	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.downloadSnapshot(RestoreClusterSnapshotMgr.java:154) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792694117Z 	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.<init>(RestoreClusterSnapshotMgr.java:53) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792699947Z 	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.init(RestoreClusterSnapshotMgr.java:67) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792705737Z 	at com.starrocks.StarRocksFE.start(StarRocksFE.java:141) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792710737Z 	at com.starrocks.StarRocksFE.main(StarRocksFE.java:102) ~[starrocks-fe.jar:?]
2026-02-24T17:18:29.792717418Z Caused by: software.amazon.awssdk.services.s3.model.S3Exception: Forbidden (Service: S3, Status Code: 403, Request ID: KHTTQA2P316NY0PG, Extended Request ID: 3hfcDh6I/GZLR59RTz2wl9zQl1nNjtZhypVA5iARUxH5NrTkH+/0c7x5ojXFvHDENlSEbgJLcKk=)
2026-02-24T17:18:29.792722488Z 	at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:104) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792727298Z 	at software.amazon.awssdk.services.s3.model.S3Exception$BuilderImpl.build(S3Exception.java:58) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792733408Z 	at software.amazon.awssdk.services.s3.internal.handlers.ExceptionTranslationInterceptor.modifyException(ExceptionTranslationInterceptor.java:88) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792738608Z 	at software.amazon.awssdk.core.interceptor.ExecutionInterceptorChain.modifyException(ExecutionInterceptorChain.java:181) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792750089Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.runModifyException(ExceptionReportingUtils.java:54) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792756109Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.utils.ExceptionReportingUtils.reportFailureToInterceptors(ExceptionReportingUtils.java:38) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792765759Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:39) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792771589Z 	at software.amazon.awssdk.core.internal.http.pipeline.stages.ExecutionFailureExceptionReportingStage.execute(ExecutionFailureExceptionReportingStage.java:26) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792777119Z 	at software.amazon.awssdk.core.internal.http.AmazonSyncHttpClient$RequestExecutionBuilderImpl.execute(AmazonSyncHttpClient.java:210) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792782570Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.invoke(BaseSyncClientHandler.java:103) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792788290Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.doExecute(BaseSyncClientHandler.java:173) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792793720Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.lambda$execute$1(BaseSyncClientHandler.java:80) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792799030Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.measureApiCallSuccess(BaseSyncClientHandler.java:182) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792804510Z 	at software.amazon.awssdk.core.internal.handler.BaseSyncClientHandler.execute(BaseSyncClientHandler.java:74) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792817331Z 	at software.amazon.awssdk.core.client.handler.SdkSyncClientHandler.execute(SdkSyncClientHandler.java:45) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792822861Z 	at software.amazon.awssdk.awscore.client.handler.AwsSyncClientHandler.execute(AwsSyncClientHandler.java:53) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792827791Z 	at software.amazon.awssdk.services.s3.DefaultS3Client.headObject(DefaultS3Client.java:7029) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792832751Z 	at software.amazon.awssdk.services.s3.DelegatingS3Client.lambda$headObject$56(DelegatingS3Client.java:5678) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792837471Z 	at software.amazon.awssdk.services.s3.internal.crossregion.S3CrossRegionSyncClient.invokeOperation(S3CrossRegionSyncClient.java:74) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792842181Z 	at software.amazon.awssdk.services.s3.DelegatingS3Client.headObject(DelegatingS3Client.java:5678) ~[bundle-2.29.52.jar:?]
2026-02-24T17:18:29.792847482Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$getObjectMetadata$10(S3AFileSystem.java:3049) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792852612Z 	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:468) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792857712Z 	at org.apache.hadoop.fs.s3a.Invoker.retryUntranslated(Invoker.java:431) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792862982Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:3036) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792868142Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.getObjectMetadata(S3AFileSystem.java:3016) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792873442Z 	at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:4079) ~[hadoop-aws-3.4.1.jar:?]
2026-02-24T17:18:29.792878593Z 	... 19 more
2026-02-24T17:18:29.794881326Z 2026-02-25 01:18:29.793+08:00 ERROR (main|1) [StarRocksFE.start():224] StarRocksFE start failed
2026-02-24T17:18:29.794918718Z com.starrocks.common.StarRocksException: Failed to copy <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532to> local /opt/starrocks/fe/meta/image
The same fe conf is working fine when I initially run the cluster and also able to insert data in the tables. But while doing the DR activity it is giving this error. ANy thought?
r
The error 
software.amazon.awssdk.services.s3.model.S3Exception: Forbidden (Status Code: 403)
indicates that the StarRocks Frontend (FE) does not have the necessary permissions to access the S3 bucket or the specific snapshot file during the restore process. Even if your initial cluster configuration works for normal operations, the Cluster Snapshot Restore process (triggered by 
RESTORE_CLUSTER_SNAPSHOT=true
) often requires an additional configuration file or specific permissions that might be missing in your DR environment. 1. Check 
cluster_snapshot.yaml
 For shared-data clusters, StarRocks uses a dedicated configuration file called 
cluster_snapshot.yaml
(located in 
fe/conf/
) to handle the restore process. This file must define the snapshot path and the storage volume credentials. Ensure this file exists in your FE pod and contains the correct credentials for the DR environment:
Copy code
yaml
cluster_snapshot:
    # The URI of the snapshot found in your logs
    cluster_snapshot_path: <s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/meta/image/automated_cluster_snapshot_1771523838532>
    storage_volume_name: my_s3_volume

storage_volumes:
  - name: my_s3_volume
    type: S3
    location: <s3://kubvir-asd21-starrocks-data/>
    properties:
      - key: aws.s3.region
        value: <your-region>
      - key: aws.s3.access_key
        value: <your-access-key>
      - key: aws.s3.secret_key
        value: <your-secret-key>
      # If using IAM roles (IRSA/Instance Profile), set the appropriate properties instead
2. Verify S3 Permissions The FE needs more than just write access. During restore, it must be able to perform: • 
s3:GetObject
• 
s3:ListBucket
• 
s3:GetBucketLocation
The error specifically occurred during 
getFileStatus
, which maps to a HeadObject or ListBucket call. Ensure your IAM policy or Bucket Policy allows these actions on the path: 
<s3://kubvir-asd21-starrocks-data/e4ed817e-0798-40a6-94b2-cd0234327463/*>
3. Kubernetes Service Account (IRSA) If you are using AWS IAM Roles for Service Accounts (IRSA): • Check if the FE pod in the DR cluster is using the same Service Account as the original cluster. • Ensure the IAM Role associated with that Service Account has a trust relationship that allows the DR EKS cluster to assume it. 4. Environment Variables If you rely on environment variables (like 
AWS_ACCESS_KEY_ID
) instead of 
fe.conf
, verify that these secrets are correctly mapped into the FE pod's environment in your DR deployment YAML. 5. Shared-Data Mode Specifics If this is a Shared-Data (Lakehouse) cluster, ensure that 
run_mode = shared_data
is set in 
fe.conf
and that the 
aws_s3_path
matches the bucket structure where the snapshot was taken. Recommendation: Try manually running a 
aws s3 ls
or 
aws s3 cp
command from within the FE pod (if the AWS CLI is installed) using the same credentials/identity to verify if the bucket is reachable. If it fails there with a 403, it is definitely an AWS-level permission or networking issue. References • administration/cluster_snapshot.md • faq/Exporting_faq.md • deployment/shared_data/s3.md
r
@Rocky I am not using aws access key and secret key in the DR activity so I added below to keys in the cluster_snapshot.yaml:
Copy code
configMaps:
  # e.g., create my-configmap
  - name: cluster-snapshot
    data:
      cluster_snapshot.yaml: |
        cluster_snapshot:
          cluster_snapshot_path: <s3://kubvir-asd21-starrocks-data/4a062057-f62b-4068-bc83-af4d6529e908/meta/image/automated_cluster_snapshot_1771951297976>
          storage_volume_name: builtin_storage_volume
        frontends: []
        compute_nodes: []
        storage_volumes:
          - name: builtin_storage_volume
            type: S3
            location: <s3://kubvir-asd21-starrocks-data>
            comment: my s3 volume
            properties:
              - key: aws.s3.region
                value: us-east-1
              - key: aws.s3.endpoint
                value: <https://s3.amazonaws.com>
              - key: aws_s3_use_instance_profile
                value: false
              - key: aws_s3_use_aws_sdk_default_behavior
                value: true
After giving S3 full access to the node role, the error is now changed to this:
Copy code
026-02-24T23:30:41+05:30 com.starrocks.common.DdlException: Invalid properties aws_s3_use_aws_sdk_default_behavior
2026-02-24T23:30:41+05:30 	at com.starrocks.server.StorageVolumeMgr.validateParams(StorageVolumeMgr.java:384) ~[starrocks-fe.jar:?]
2026-02-24T23:30:41+05:30 	at com.starrocks.server.StorageVolumeMgr.replaceStorageVolume(StorageVolumeMgr.java:222) ~[starrocks-fe.jar:?]
2026-02-24T23:30:41+05:30 	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.updateStorageVolumes(RestoreClusterSnapshotMgr.java:249) ~[starrocks-fe.jar:?]
2026-02-24T23:30:41+05:30 	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.finishRestoring(RestoreClusterSnapshotMgr.java:92) ~[starrocks-fe.jar:?]
2026-02-24T23:30:41+05:30 	at com.starrocks.StarRocksFE.start(StarRocksFE.java:213) ~[starrocks-fe.jar:?]
2026-02-24T23:30:41+05:30 	at com.starrocks.StarRocksFE.main(StarRocksFE.java:102) ~[starrocks-fe.jar:?]
r
Hello! I'm Rocky. It looks like you're encountering a configuration error because of the naming convention used for Storage Volume properties. The error 
Invalid properties aws_s3_use_aws_sdk_default_behavior
occurs because Storage Volume properties in StarRocks must use the dot (
.
) notation, whereas FE configuration parameters (usually found in 
fe.conf
) use underscores (
_
). In your 
cluster_snapshot.yaml
, you have correctly used dots for 
aws.s3.region
and 
aws.s3.endpoint
, but you used underscores for the credential-related properties. Recommended Fix Update your 
storage_volumes
section to use dots for all keys. Additionally, if you are using a node role (Instance Profile) or IRSA, you typically set 
aws.s3.use_instance_profile
to 
true
. Here is the corrected snippet for your `cluster_snapshot.yaml`:
Copy code
yaml
        storage_volumes:
          - name: builtin_storage_volume
            type: S3
            location: <s3://kubvir-asd21-starrocks-data>
            comment: my s3 volume
            properties:
              - key: aws.s3.region
                value: us-east-1
              - key: aws.s3.endpoint
                value: <https://s3.amazonaws.com>
              - key: aws.s3.use_instance_profile
                value: true
              - key: aws.s3.use_aws_sdk_default_behavior
                value: true
Key Changes: 1. 
aws_s3_use_instance_profile
 $\rightarrow$ 
aws.s3.use_instance_profile
: Changed underscores to dots. Also, set this to 
true
if you want StarRocks to use the IAM role attached to your K8s nodes or the IAM Role for Service Accounts (IRSA). 2. 
aws_s3_use_aws_sdk_default_behavior
 $\rightarrow$ 
aws.s3.use_aws_sdk_default_behavior
: Changed underscores to dots. 3. Remove Angle Brackets: Ensure that the 
location
and 
endpoint
do not contain the 
< >
brackets in the actual configuration (e.g., 
value: <https://s3.amazonaws.com>
instead of 
value: <https://s3.amazonaws.com>
). Why this happened? StarRocks distinguishes between global FE configurations and Storage Volume properties: * FE Config (
fe.conf
): Uses 
aws_s3_use_aws_sdk_default_behavior
. * Storage Volume Property: Uses 
aws.s3.use_aws_sdk_default_behavior
. Since the Disaster Recovery activity (Cluster Snapshot) involves creating/replacing storage volumes, it expects the property names to match the Storage Volume schema. References • administration/management/FE_configuration.md • deployment/shared_data/s3.md • sql-reference/sql-statements/cluster-management/storage_volume/CREATE_STORAGE_VOLUME.md
r
@Rocky I want to only use IRSA and not the node role while doing DR activity
Copy code
- key: aws.s3.use_instance_profile
                value: true
              - key: aws.s3.use_aws_sdk_default_behavior
                value: true
r
To use IRSA (IAM Roles for Service Accounts) specifically and avoid falling back to the EC2 Node/Instance Role during DR (Disaster Recovery) activities or general S3 operations, you should focus on the 
aws.s3.use_aws_sdk_default_behavior
parameter. In an EKS environment, IRSA works by injecting environment variables (
AWS_WEB_IDENTITY_TOKEN_FILE
and 
AWS_ROLE_ARN
) into the Pod. The AWS SDK's Default Credentials Provider Chain automatically detects these variables and uses them for authentication. Recommended Configuration For your specific requirement to use IRSA:
Copy code
yaml
- key: aws.s3.use_aws_sdk_default_behavior
  value: "true"
- key: aws.s3.use_instance_profile
  value: "false"
Why this configuration? 1. 
aws.s3.use_aws_sdk_default_behavior: true
: This tells StarRocks to use the standard AWS SDK credential search order. In EKS, the Web Identity Token (IRSA) is checked before the Instance Profile (Node Role). This is the primary way to enable IRSA. 2. 
aws.s3.use_instance_profile: false
: Setting this to 
false
(or omitting it) ensures that StarRocks does not explicitly attempt to use the EC2 metadata service for credentials outside of the default SDK chain. Note that in the default SDK chain, the instance profile is the last resort. Key Steps for IRSA to Work: 1. ServiceAccount: Ensure your StarRocks FE/BE pods are using a Kubernetes 
ServiceAccount
that is annotated with the IAM Role ARN:
Copy code
yaml
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      annotations:
        <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: arn:aws:iam::123456789012:role/your-irsa-role
      name: starrocks-sa
2. Pod Specification: The pods must be configured to use this 
ServiceAccount
. EKS will then automatically inject the necessary identity tokens. 3. Trust Relationship: The IAM Role must have a trust relationship allowing the OIDC provider of your EKS cluster to assume the role. DR Specific Tip: If you are performing a Backup or Restore for DR, ensure that the 
BACKUP
or 
RESTORE
command itself uses these properties if they aren't globally defined in the FE configuration. For example:
Copy code
sql
BACKUP SNAPSHOT db_name TO backup_repo
PROPERTIES (
    "aws.s3.use_aws_sdk_default_behavior" = "true"
);
Warning: If you set 
aws.s3.use_instance_profile: true
along with the default behavior, the system might fall back to the Node Role if the IRSA token is missing or expired. To strictly enforce IRSA, rely on the 
use_aws_sdk_default_behavior
and ensure your IAM policies on the Node Role are restricted. References • integrations/authenticate_to_aws_resources.md • deployment/shared_data/s3.md
r
@Rocky What are the recommended metrics to show on grafana and alerts for CN nodes?
r
For StarRocks Compute Nodes (CN), monitoring focuses on computational efficiency, memory management, and cache performance (since CNs are primarily used in shared-data/cloud-native architectures). Here are the recommended metrics and alert rules for your Grafana dashboard: 1. Node Health & Resource Utilization These metrics ensure the CN process is running and has enough resources. * Node Status (
up
): Monitors if the CN process is alive. * CPU Usage: 
starrocks_be_cpu_usage_percent
. _ _Alert:* CPU > 80% for 5 minutes. * Memory Usage: 
starrocks_be_memory_allocated_bytes
. CNs use memory for query execution and metadata. _ _Alert:* Memory usage > 90% of the 
mem_limit
. * File Descriptor Limit: 
starrocks_be_fd_num_used
vs 
starrocks_be_fd_num_limit
. _ _Alert:* FD usage > 80%. 2. Data Cache Metrics (Crucial for CN) Since CNs pull data from remote storage (S3, HDFS, etc.), the local Data Cache is vital for performance. * Cache Hit Rate: 
starrocks_be_data_cache_read_block_count
vs 
starrocks_be_data_cache_read_block_cache_hit_count
. _ _Alert:* Hit rate < 50% (depending on your workload baseline). * Cache Disk Usage: Monitor the disk space of the directory used for 
starrocks_be_data_cache_metrics
. _ _Alert:* Cache disk usage > 90%. * Remote Read Throughput: 
starrocks_be_data_cache_read_remote_throughput_bytes
. Monitors how much data is being pulled from object storage. 3. Query Execution & Performance * Query QPS: 
starrocks_be_query_scan_rows
and throughput to see the load handled by the CN. * Query Latency: P99/P95 latency of queries executed on the CN. * RPC Latency/Errors: 
starrocks_be_rpc_latency_ms
and 
starrocks_be_rpc_error_count
. Monitors communication between FE and CN. _ _Alert:* Significant spike in RPC errors. 4. Recommended Alert Rules (Prometheus/Grafana) Alert Name | Expression (Simplified) | Severity CN Node Down | 
up{job="starrocks_cn"} == 0
| Critical High CPU Usage | 
starrocks_be_cpu_usage_percent > 85
| Warning Memory Exhaustion | 
starrocks_be_memory_allocated_bytes / starrocks_be_memory_limit_bytes > 0.9
| Critical Data Cache Low Hit Rate | 
(hit_count / total_count) < 0.4
| Warning High RPC Error Rate | 
rate(starrocks_be_rpc_error_count[5m]) > 10
| Warning/Critical How to set up: 1. Prometheus StarRocks Exporter: Ensure your CNs are configured to export metrics (usually on port 
8040
at 
/metrics
). 2. Official Dashboard: It is highly recommended to use the StarRocks Grafana Dashboard available in the StarRocks GitHub repository or via the official documentation. These templates often have a "Compute Node" or "BE/CN" toggle. Rocky's Tip: In a cloud-native (shared-data) deployment, keep a close eye on Remote Read Throughput. If it spikes consistently while the Cache Hit Rate is low, you might need to increase the local disk space allocated for the Data Cache on your CN nodes.
r
@Rocky Not seeing these metrics in prometheus for CN nodes: 
starrocks_be_memory_allocated_bytes
starrocks_be_data_cache_read_block_count
starrocks_be_data_cache_read_block_cache_hit_count
r
Hello! I'm Rocky. I can help you troubleshoot why those specific metrics might be missing for your CN (Compute Node) nodes in Prometheus. Even though they are CN nodes, StarRocks typically uses the 
starrocks_be_
prefix for backend-related metrics since CNs share the same core engine as BEs. Here are the most common reasons why these metrics might be missing: 1. Data Cache Metrics (
data_cache_read_block_...
) These metrics only appear and populate if the Data Cache feature is enabled. Data Cache is used primarily for accelerating queries on external tables (like S3/HDFS) or when running in shared-data mode. * Check Configuration: Ensure that 
enable_data_cache = true
is set in your 
cn.conf
. * Initialization: In some versions, these metrics might not show up in the 
/metrics
endpoint until the Data Cache is initialized or the first query utilizing the cache has been executed. * Verify Feature: If you are not using external tables or shared-data mode, the Data Cache might not be active, which would explain why these metrics are absent. 2. Memory Metric (
memory_allocated_bytes
) This is a standard memory metric. If you see other 
starrocks_be_
metrics but not this one: * StarRocks Version: Check your StarRocks version. Some memory-specific metrics were renamed or introduced in newer versions (e.g., 2.5 or 3.x). You might want to look for 
starrocks_be_process_mem_bytes
or 
starrocks_be_query_mem_bytes
as alternatives depending on what you are trying to measure. * Direct Check: Run a curl command directly against the CN node's web port (default 
8040
) to see if the metric exists in the raw output:
Copy code
bash
    curl http://<CN_IP>:8040/metrics | grep starrocks_be_memory_allocated_bytes
3. General Troubleshooting for CN Metrics * Prometheus Scrape Config: Ensure your Prometheus is actually scraping the CN nodes. If you see 
starrocks_fe_
metrics but no 
starrocks_be_
metrics for your CN IP addresses, the CN scrape job might be misconfigured. * Port Mapping: Ensure the CN nodes are reachable on their heart-beat or web port (default 
8040
) by the Prometheus server. * Labels: Sometimes metrics might be categorized under different labels. Check if the metrics exist but are associated with a 
job
or 
instance
label that you didn't expect. Quick tip: If you are using the StarRocks Kubernetes Operator, ensure the 
ServiceMonitor
is correctly picking up the CN pods. If you can provide your StarRocks version and confirm if 
enable_data_cache
is set to 
true
, I can provide more specific guidance!
2 Views
Open in Slack
PreviousNext
Powered by