http://starrocks.io logo
Join Slack
Powered by
:sos: Hi team, I'm trying to recover my cluster us...
# questions-and-troubleshooting
a
🆘 Hi team, I'm trying to recover my cluster using disaster recovery mode. I created the 
cluster_snapshot.yaml
and the required ConfigMaps, and then attempted to recreate the cluster. However, the FE node keeps crashing. Has anyone encountered this before or knows how to fix it? @Rocky 
cluster_snapshot.yaml
Copy code
# cluster_snapshot.yaml
starrocks:
  starrocksCluster:
    disasterRecovery:
      enabled: true
      configMapName: cluster-snapshot
      generation: 1     # increment to retry recovery

  starrocksFESpec:
    # mount cluster_snapshot.yaml into the FE conf directory
    configMaps:
      - name: cluster-snapshot
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml
configMaps
Copy code
cluster_snapshot:
  cluster_snapshot_path: <abfss://test-recovery>@ststarrockwus01.dfs.core.windows.net/shared_data/057ba641-a540-4eaf-9e08-8d40e28e0aaf/meta/image/automated_cluster_snapshot_1777459778177
  storage_volume_name: builtin_storage_volume

frontends: []
compute_nodes: []

storage_volumes:
  - name: builtin_storage_volume
    type: ADLS2
    location: <abfss://test-recovery>@ststarrockwus01.dfs.core.windows.net/shared_data
    comment: primary ADLS volume
    properties:
      - key: azure.adls2.endpoint
        value: <https://ststarrockwus01.dfs.core.windows.net>
      - key: azure.adls2.shared_key
        value: 06lkNos5W44+qJLhvqxvzKYkoOkJ
r
Hello! I'm Rocky. It looks like you're running into issues with the StarRocks disaster recovery (DR) mode. This usually happens due to configuration mismatches or credential issues that prevent the Leader FE from initializing the recovery process. Based on the snippets you provided, here are the most likely reasons for the FE crash and how to fix them: 1. Remove Angle Brackets from URIs The angle brackets (
< >
) in your 
cluster_snapshot_path
, 
location
, and 
endpoint
should not be there. In StarRocks configuration files, these are usually placeholders in templates. They will cause parsing or connection errors if included in the actual YAML. Corrected URI format:
Copy code
yaml
cluster_snapshot:
  cluster_snapshot_path: <abfss://test-recovery@ststarrockwus01.dfs.core.windows.net/shared_data/>...
  storage_volume_name: builtin_storage_volume

storage_volumes:
  - name: builtin_storage_volume
    type: ADLS2
    location: <abfss://test-recovery@ststarrockwus01.dfs.core.windows.net/shared_data>
    properties:
      - key: azure.adls2.endpoint
        value: <https://ststarrockwus01.dfs.core.windows.net>
2. Verify the Azure Shared Key The 
azure.adls2.shared_key
you provided (
06lkNos5W44+qJLhvqxvzKYkoOkJ
) appears significantly shorter than a standard Azure Storage Account Key, which is typically 88 characters. If the key is invalid or truncated, the FE will crash as soon as it tries to authenticate with Azure to pull the metadata. 3. Clean FE Metadata (Crucial) Disaster recovery requires starting the Leader FE with a completely empty metadata directory. * If you are recreating the cluster in the same namespace with the same name, make sure to delete the existing FE PVCs before applying the new 
StarRocksCluster
spec. * If the FE finds old metadata, it will ignore the 
cluster_snapshot.yaml
or crash due to state inconsistency. 4. Redundant ConfigMap Mounting In the StarRocks Operator (v1.10+), if you set 
disasterRecovery.enabled: true
and specify 
configMapName
, the operator handles the mounting and the 
--cluster_snapshot
start flag for you. * You generally do not need to manually add the ConfigMap to 
starrocksFESpec.configMaps
unless you are on a very old version of the operator. * Double-check that the file is actually named 
cluster_snapshot.yaml
inside the ConfigMap (the key in the ConfigMap should match the filename). 5. Check the FE Logs To find the exact reason for the crash, check the FE container logs:
Copy code
bash
kubectl logs <fe-pod-name> -c starrocks-fe
Look for: * `java.lang.IllegalArgumentException`: Usually indicates a URI parsing error (check the brackets). * `AccessDenied`: Credential or permission issue with ADLS2. * `Frontend leader startup failed`: Check the trace above this line for the specific root cause. Summary Checklist: • ☐ Remove all 
<
and 
>
brackets from paths and endpoints. • ☐ Ensure the Azure Shared Key is the full 88-character string. • ☐ Delete old FE PVCs to ensure a clean 
meta
directory. • ☐ Ensure the 
storage_volume_name
matches exactly what was used in the original cluster. If the logs show a specific error message, feel free to share it here! References • administration/cluster_snapshot.md • administration/Meta_recovery.md
a
The command I used to deploy the cluster
Copy code
kubectl create configmap cluster-snapshot \
  -n dr-starrocks-ctr-t2 \
  --from-file=cluster_snapshot.yaml=./snapshot-content.yaml
Copy code
helm install starrocks ./starrocks-cn/kube-starrocks -n dr-starrocks-ctr-t2 --create-namespace -f custom-values-dev.yaml -f cluster_snapshot.yaml
n
@Arun Krishna I dont think there is support for ADLS yet. @Rocky
r
You are absolutely right to be skeptical. While StarRocks added support for ADLS2 as a storage backend for shared-data clusters in v3.4.1, there is a known limitation regarding the Cluster Snapshot (Disaster Recovery) feature on Azure. The Problem: Scheme Mismatch The issue is that while the CN (Compute Nodes) use a native StarRocks implementation to talk to ADLS2 (using the 
adls2://
or 
azblob://
schemes), the FE (Frontend) uses the Hadoop FileSystem API to upload and download the metadata snapshots. 1. Hadoop doesn't know 
adls2://
: The Hadoop FileSystem API used by the FE does not recognize the StarRocks-specific 
adls2://
scheme. 2. Hadoop expects 
abfss://
: For ADLS2, Hadoop expects the 
abfss://
scheme. 3. Property Passing: Even if you change the scheme to 
abfss://
, the StarRocks FE (in current versions up to 3.4.x) may not correctly pass the Azure credentials (like the shared key) from the 
storage_volume
configuration to the underlying Hadoop environment. Why your FE is crashing Based on your YAML, here are the specific triggers for the crash: * Invalid URIs: You still have angle brackets 
< >
in your paths (e.g., 
<abfss://test-recovery>
). These will cause the FE to throw a 
java.net.URISyntaxException
or 
IllegalArgumentException
immediately. * Unsupported Scheme: If you use 
type: ADLS2
, the FE attempts to resolve it using logic that may not be fully wired for Hadoop-based snapshots yet, leading to an 
UnsupportedFileSystemException
. Is there a workaround? Currently, there isn't a "native" way to use the 
ADLS2
storage volume type for the 
cluster_snapshot
feature. However, you can try to "trick" the FE by defining the storage volume as type: HDFS in your 
cluster_snapshot.yaml
, which forces the FE to use the standard Hadoop Azure (ABFS) driver. Experimental Workaround: Try modifying your 
cluster_snapshot.yaml
to look like this (ensuring no 
< >
and using the full Azure Key):
Copy code
yaml
cluster_snapshot:
  cluster_snapshot_path: <abfss://test-recovery@ststarrockwus01.dfs.core.windows.net/shared_data/057ba641>...
  storage_volume_name: builtin_storage_volume

storage_volumes:
  - name: builtin_storage_volume
    type: HDFS
    location: <abfss://test-recovery@ststarrockwus01.dfs.core.windows.net/shared_data>
    properties:
      - key: <http://fs.azure.account.key.ststarrockwus01.dfs.core.windows.net|fs.azure.account.key.ststarrockwus01.dfs.core.windows.net>
        value: "YOUR_FULL_88_CHARACTER_SECRET_KEY"
Important Notes: * Azure JARs: This workaround only works if the StarRocks FE image includes the 
hadoop-azure
and 
azure-storage
JAR files (which are usually included in recent versions). * Version Check: Ensure you are on v3.4.2 or higher, as earlier versions do not support the 
cluster_snapshot
feature at all. * Official Support: Official, seamless support for 
type: ADLS2
in snapshots is a high-priority item for the engineering team and is expected to be refined in upcoming releases. If the FE still crashes, please check the logs (
kubectl logs <fe-pod>
) and look for the specific Java stack trace. If it says 
UnsupportedFileSystemException: No FileSystem for scheme: adls2
, it confirms the scheme mismatch. References • deployment/shared_data/feature-support-shared-data.md • administration/cluster_snapshot.md
Open in Slack
PreviousNext
Powered by