http://starrocks.io logo
Join Slack
Powered by
:rotating_light: SOS - HIGH PRIORITY StarRocks DR ...
# questions-and-troubleshooting
a
🚨 SOS - HIGH PRIORITY StarRocks DR install failing — need help @Rocky Hi team, blocked on a StarRocks disaster recovery deployment. Any guidance appreciated Setup: • Chart: 
kube-starrocks
(shared-data mode, ADLS2 backend) • Namespace: 
dr-starrocks-ctr-t4
• Goal: restore cluster from 
automated_cluster_snapshot
on ADLS2 Issue: Fresh 
helm install
with 
custom-values-dev.yaml
+ 
cluster_snapshot.yaml
→ FE pod (
kube-starrocks-fe-0
) crashes repeatedly with 
Error
status (3 restarts in ~2.5 min), even though 
startupProbeFailureSeconds: 300
is set. FE logs end with:
Copy code
FE service kube-starrocks-fe-service:9030 is not alive yet!
Timed out, no members detected ever, assume myself is the first node ..
first start with no meta run start_fe.sh with additional options: ' --host_type FQDN'
Operator status shows DR is engaged:
Copy code
disasterRecoveryStatus:
  observedGeneration: 4
  phase: doing
  reason: disaster recovery is in progress
What I've checked: • Mounted 
cluster_snapshot.yaml
in the FE pod is the default commented template, not my configmap content → suspect mount path collision with chart's conf volume • 
helm upgrade
(without DR file) somehow succeeds and restores all DBs/tables/MVs/data → but install fails • Chart values.yaml note says 
startupProbeFailureSeconds
requires CRD v1.8.7+ — kubectl get crd starrocksclusters.starrocks.com -o yaml | grep -A2 "labels\|annotations" | head -20 annotations: controller-gen.kubebuilder.io/version: v0.14.0 version: v1.11.2 -- annotations: additionalProperties: type: string -- annotations: additionalProperties: type: string -- labels: additionalProperties: type: string -- annotations: additionalProperties: type: string -- Questions: 1. Anyone seen FE crash on first DR install but recover on upgrade? 2. Is the 
configMaps:
top-level (chart-managed) pattern required, vs pre-creating the configmap with 
kubectl create configmap
? 3. What's the minimum CRD/operator version for 
disasterRecovery.enabled
to actually work? Happy to share full values files / pod describe output in thread 🧵
custom-values-dev.yamlcluster_snapshot.yaml
r
Hello! Rocky here. It sounds like you're hitting a specific "chicken-and-egg" problem with the StarRocks Operator's Disaster Recovery (DR) mode and how the 
cluster_snapshot.yaml
is being projected into the pod. Based on the logs and symptoms you've shared, here is the breakdown of what's likely happening and how to fix it: 1. The "Default Template" & Mount Path Collision You mentioned that the 
cluster_snapshot.yaml
inside the pod is the default template. This is the root cause of the crash. When the FE starts in DR mode, it requires a valid 
cluster_snapshot_path
. If it sees a template with commented-out or dummy values, the FE process will fail to initialize. * The Issue: If you mount a ConfigMap to 
/opt/starrocks/fe/conf/
without using 
subPath
, it will overwrite the entire configuration directory. If you mount it specifically to the file path but without 
subPath
, Kubernetes might create it as a directory. * The Fix: You must use the 
subPath
field in your 
starrocksFESpec.configMaps
to ensure you are only overriding that specific file and not the whole directory or getting blocked by the image's default file. Correct 
values.yaml
snippet:
Copy code
yaml
starrocksCluster:
  starrocksFESpec:
    configMaps:
      - name: cluster-snapshot-config  # Name of your ConfigMap
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml
2. Why is the FE starting without the DR flag? Your logs show: 
first start with no meta run start_fe.sh with additional options: ' --host_type FQDN'
. Notice that 
--cluster_snapshot
is missing from the options. * Operator Logic: For the DR process to trigger, the StarRocks Operator must inject the 
--cluster_snapshot
flag into the container's startup arguments. * The Fix: Ensure your 
StarRocksCluster
(SRC) spec actually contains the DR configuration. Run 
kubectl get starrockscluster <name> -o yaml
and verify that 
spec.disasterRecovery.enabled
is 
true
. * Generation Field: The operator uses the 
generation
field to trigger a new DR attempt. If you are re-running an install or retry, try incrementing the 
generation
value (e.g., from 
1
to 
2
) in your YAML. 3. Minimum Version Requirements * Operator/CRD: You are on 
v1.11.2
, which is well above the v1.10.0 minimum required for the 
disasterRecovery
field to be functional. * StarRocks Version: For the 
automated_cluster_snapshot
feature in shared-data mode (ADLS2), you generally need StarRocks v3.5.0 or later. If your image is older (e.g., v3.1 or v3.2), the binary may not support restoring from these automated snapshots via the YAML config. 4. Crash on Install vs. Success on Upgrade * Why it fails on first install: The Operator's reconciliation loop for a "fresh" cluster might be trying to deploy the full FE spec before the DR status is fully acknowledged, or the startup probe (even at 300s) is killing the pod because the script is stuck waiting for a service that won't start until the snapshot is processed. * Why it works on upgrade: On an upgrade, the 
ConfigMap
and the 
SRC
spec (with DR enabled) are already present in the K8s API. The Operator can generate the correct Pod spec with the 
--cluster_snapshot
flag from the very first container start. Summary Checklist for Success: 1. Check StarRocks Version: Ensure you are using StarRocks v3.5+ for shared-data snapshot DR. 2. Verify Mount: Use 
kubectl exec
into the crashing pod (if it stays up long enough) or check a 
describe pod
to ensure the 
cluster-snapshot
volume is actually using a 
subPath
. 3. ConfigMap Pattern: It is safer to use the top-level chart-managed 
configMaps:
pattern in 
kube-starrocks
. This ensures the ConfigMap is created before the Operator tries to mount it to the FE.
4. Clean State: If you've had failed attempts, ensure you delete the FE metadata PVCs in the namespace 
dr-starrocks-ctr-t4
before the fresh 
helm install
. DR mode will not trigger correctly if the operator detects existing metadata on the disk. Pro Tip: If the pod keeps crashing too fast to debug, temporarily set the FE 
terminationGracePeriodSeconds
higher and check 
kubectl logs -p
(previous) to see if there are Java-level errors about the ADLS2 credentials inside the 
fe.log
. References • administration/cluster_snapshot.md • administration/Meta_recovery.md • quick_start/helm.md
Hello! Rocky here. It looks like you're caught in a common "chicken-and-egg" cycle with the StarRocks Operator's Disaster Recovery (DR) mode. The fact that your FE logs show 
--host_type FQDN
but miss the 
--cluster_snapshot
flag, combined with the FE seeing a "default template" file, points to a configuration projection issue. Here are the answers to your specific questions and a troubleshooting guide: 1. FE Crash on Install vs. Success on Upgrade The reason 
helm install
fails while 
upgrade
(sometimes) works is often related to the Operator's state machine. * On Install: The Operator is trying to provision the FE for the first time. If the DR state (
spec.disasterRecovery
) is present from the start, the Operator enters the 
todo
-> 
doing
phase. However, if the FE Pod doesn't see the correct 
cluster_snapshot.yaml
(due to the mount issue), it crashes. * On Upgrade: If you have already run a failed install, the PVCs for FE metadata might already exist. If you then upgrade, the FE might be trying to start using "half-baked" or partial metadata from the previous attempt. If it "succeeds" and shows data, it means the FE metadata was actually successfully restored or preserved in the local volume at some point, and it's no longer in "fresh start" mode. * Warning: In shared-data mode, if you want a clean DR restore, you must delete the FE metadata PVCs before the install/restore. Otherwise, the FE will try to join an existing (but broken) cluster instead of restoring from the snapshot. 2. ConfigMap Pattern: Chart-managed vs. Pre-created While 
kubectl create configmap
works, the top-level 
configMaps:
pattern in 
kube-starrocks
 is strongly recommended for DR for two reasons: 1. Consistency: It ensures the ConfigMap is created and synchronized before the 
StarRocksCluster
(SRC) object is processed. 2. Mount Management: The chart handles the volume and volumeMount definitions automatically based on your 
starrocksFESpec.configMaps
entries. Crucial Fix for the "Default Template" issue: The StarRocks image contains a template file at 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
. If you mount your ConfigMap without a 
subPath
, Kubernetes may either fail the mount or overwrite the directory. You must use 
subPath
. Example 
values.yaml
snippet:
Copy code
yaml
starrocksCluster:
  disasterRecovery:
    enabled: true
    generation: 1 # Increment this to trigger a retry
  starrocksFESpec:
    configMaps:
      - name: dr-snapshot-conf
        mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
        subPath: cluster_snapshot.yaml # This prevents collision with the image file
3. Minimum Versions * Operator: You are on 
v1.11.2
, which is perfect. DR support was introduced in v1.10.0. * StarRocks: Restoration from 
automated_cluster_snapshot
in shared-data mode requires StarRocks v3.4.2 or later (v3.5+ recommended for ADLS2 stability). Why the 
--cluster_snapshot
flag is missing: The Operator only injects the 
--cluster_snapshot
argument into the FE startup command when: 1. 
spec.disasterRecovery.enabled
is 
true
. 2. The 
disasterRecoveryStatus.phase
is 
todo
or 
doing
. 3. The FE metadata PVC is empty. If the Operator detects existing metadata, it assumes the cluster is already initialized and will skip the DR bootstrap flags. Recommended SOS Recovery Steps: 1. Clean the Slate: Delete the current helm release and manually delete the FE metadata PVCs in namespace 
dr-starrocks-ctr-t4
. 2. Verify the Snapshot Path: Ensure the 
cluster_snapshot_path
in your ConfigMap points to the exact folder containing the 
image
(e.g., 
.../meta/image/automated_cluster_snapshot_12345
). 3. Use `subPath`: Update your 
custom-values-dev.yaml
to include the 
subPath
as shown above. 4. Fresh Install: Run your 
helm install
.
5. Check the Spec: Run 
kubectl get src -n dr-starrocks-ctr-t4 -o yaml
and verify that 
spec.disasterRecovery.enabled
is truly 
true
in the resulting object. If the flag is still missing, check the Operator logs (
kubectl logs -n <operator-namespace> ...
). It will explicitly log why it is skipping the DR injection if it sees an issue with the PVCs or the spec. the metadata volume. References • Cluster Snapshot Documentation • Operator DR Guide operator)
a
From fe.warn.log:
ERROR StarRocksFE start failed com.starrocks.common.DdlException: Invalid properties azure_path_key at StorageVolumeMgr.validateParams(StorageVolumeMgr.java:386) at StorageVolumeMgr.replaceStorageVolume(StorageVolumeMgr.java:224) at RestoreClusterSnapshotMgr.updateStorageVolumes(RestoreClusterSnapshotMgr.java:254) at RestoreClusterSnapshotMgr.finishRestoring(RestoreClusterSnapshotMgr.java:93) at StarRocksFE.start(StarRocksFE.java:206)
Copy code
root@kube-starrocks-fe-0:/opt/starrocks/fe/log# head -n 5000 fe.log
2026-05-07 14:48:08.727+08:00 DEBUG (main|1) [LoggingProviderImpl$JULWrapper.log():274] SecurityPropertyModification: key:networkaddress.cache.ttl, value:60
2026-05-07 14:48:08.733+08:00 INFO (main|1) [RestoreClusterSnapshotMgr.init():67] FE start to restore from a cluster snapshot (RESTORE_CLUSTER_SNAPSHOT=true)
2026-05-07 14:48:08.942+08:00 INFO (main|1) [RestoreClusterSnapshotMgr.downloadSnapshot():155] Download cluster snapshot <adls2://drstarrockstest/shared_data/3c76279d-4fd6-40aa-96b5-717032118104/meta/image/automated_cluster_snapshot_1778135089752> to local dir /opt/starrocks/fe/meta/image
2026-05-07 14:48:09.002+08:00 INFO (main|1) [HdfsFsManager.getFileSystemByCloudConfiguration():758] could not find file system for path <abfss://drstarrockstest@ststarrockssvcdevccwus01.dfs.core.windows.net/shared_data/3c76279d-4fd6-40aa-96b5-717032118104/meta/image/automated_cluster_snapshot_1778135089752> create a new one
2026-05-07 14:48:09.089+08:00 DEBUG (main|1) [Shell.<clinit>():573] Failed to detect a valid hadoop home directory
java.io.FileNotFoundException: HADOOP_HOME and hadoop.home.dir are unset.
	at org.apache.hadoop.util.Shell.checkHadoopHomeInner(Shell.java:521) ~[hadoop-common-3.4.3.jar:?]
	at org.apache.hadoop.util.Shell.checkHadoopHome(Shell.java:492) ~[hadoop-common-3.4.3.jar:?]
	at org.apache.hadoop.util.Shell.<clinit>(Shell.java:569) ~[hadoop-common-3.4.3.jar:?]
	at org.apache.hadoop.util.StringUtils.<clinit>(StringUtils.java:80) ~[hadoop-common-3.4.3.jar:?]
	at org.apache.hadoop.conf.Configuration.getBoolean(Configuration.java:1733) ~[hadoop-common-3.4.3.jar:?]
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:598) ~[starrocks-hadoop-ext.jar:?]
	at com.starrocks.fs.hdfs.HdfsFsManager.getFileSystemByCloudConfiguration(HdfsFsManager.java:791) ~[starrocks-fe.jar:?]
	at com.starrocks.fs.hdfs.HdfsFsManager.getAzureFileSystem(HdfsFsManager.java:723) ~[starrocks-fe.jar:?]
	at com.starrocks.fs.hdfs.HdfsFsManager.getFileSystem(HdfsFsManager.java:475) ~[starrocks-fe.jar:?]
	at com.starrocks.fs.hdfs.HdfsFsManager.copyToLocal(HdfsFsManager.java:1215) ~[starrocks-fe.jar:?]
	at com.starrocks.fs.hdfs.HdfsService.copyToLocal(HdfsService.java:58) ~[starrocks-fe.jar:?]
	at com.starrocks.fs.HdfsUtil.copyToLocal(HdfsUtil.java:71) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.downloadSnapshot(RestoreClusterSnapshotMgr.java:156) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.<init>(RestoreClusterSnapshotMgr.java:54) ~[starrocks-fe.jar:?]
	at com.starrocks.lake.snapshot.RestoreClusterSnapshotMgr.init(RestoreClusterSnapshotMgr.java:68) ~[starrocks-fe.jar:?]
	at com.starrocks.StarRocksFE.start(StarRocksFE.java:134) ~[starrocks-fe.jar:?]
	at com.starrocks.StarRocksFE.main(StarRocksFE.java:95) ~[starrocks-fe.jar:?]
2026-05-07 14:48:09.114+08:00 DEBUG (main|1) [Shell.isSetsidSupported():874] setsid exited with exit code 0
2026-05-07 14:48:09.114+08:00 DEBUG (main|1) [FileSystem.get():599] Bypassing cache to create filesystem <abfss://drstarrockstest@ststarrockssvcdevccwus01.dfs.core.windows.net/shared_data/3c76279d-4fd6-40aa-96b5-717032118104/meta/image/automated_cluster_snapshot_1778135089752>
2026-05-07 14:48:09.115+08:00 INFO (main|1) [FileSystem.createFileSystemInternal():3735] [hadoop-ext] FileSystem.createFileSystem
2026-05-07 14:48:09.120+08:00 DEBUG (main|1) [DurationInfo.<init>():80] Starting: Creating FS <abfss://drstarrockstest@ststarrockssvcdevccwus01.dfs.core.windows.net/shared_data/3c76279d-4fd6-40aa-96b5-717032118104/meta/image/automated_cluster_snapshot_1778135089752>
2026-05-07 14:48:09.121+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3644] Loading filesystems
2026-05-07 14:48:09.126+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] nullscan:// = class org.apache.hadoop.hive.ql.io.NullScanFileSystem from /opt/starrocks/fe/lib/hive-apache-3.1.2-22.jar
2026-05-07 14:48:09.144+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] file:// = class org.apache.hadoop.fs.LocalFileSystem from /opt/starrocks/fe/lib/hadoop-common-3.4.3.jar
2026-05-07 14:48:09.145+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] file:// = class org.apache.hadoop.hive.ql.io.ProxyLocalFileSystem from /opt/starrocks/fe/lib/hive-apache-3.1.2-22.jar
2026-05-07 14:48:09.158+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] hdfs:// = class org.apache.hadoop.hdfs.DistributedFileSystem from /opt/starrocks/fe/lib/hadoop-hdfs-client-3.4.3.jar
2026-05-07 14:48:09.177+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] webhdfs:// = class org.apache.hadoop.hdfs.web.WebHdfsFileSystem from /opt/starrocks/fe/lib/hadoop-hdfs-client-3.4.3.jar
2026-05-07 14:48:09.181+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] swebhdfs:// = class org.apache.hadoop.hdfs.web.SWebHdfsFileSystem from /opt/starrocks/fe/lib/hadoop-hdfs-client-3.4.3.jar
2026-05-07 14:48:09.229+08:00 DEBUG (main|1) [MutableMetricsFactory.newForField():43] field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.getGroups with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"GetGroups"})
2026-05-07 14:48:09.231+08:00 DEBUG (main|1) [MutableMetricsFactory.newForField():43] field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginFailure with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Rate of failed kerberos logins and latency (milliseconds)"})
2026-05-07 14:48:09.231+08:00 DEBUG (main|1) [MutableMetricsFactory.newForField():43] field org.apache.hadoop.metrics2.lib.MutableRate org.apache.hadoop.security.UserGroupInformation$UgiMetrics.loginSuccess with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Rate of successful kerberos logins and latency (milliseconds)"})
2026-05-07 14:48:09.232+08:00 DEBUG (main|1) [MutableMetricsFactory.newForField():43] field private org.apache.hadoop.metrics2.lib.MutableGaugeInt org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailures with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Renewal failures since last successful login"})
2026-05-07 14:48:09.232+08:00 DEBUG (main|1) [MutableMetricsFactory.newForField():43] field private org.apache.hadoop.metrics2.lib.MutableGaugeLong org.apache.hadoop.security.UserGroupInformation$UgiMetrics.renewalFailuresTotal with annotation @org.apache.hadoop.metrics2.annotation.Metric(always=false, sampleName="Ops", valueName="Time", about="", interval=10, type=DEFAULT, value={"Renewal failures since startup"})
2026-05-07 14:48:09.234+08:00 DEBUG (main|1) [MetricsSystemImpl.register():231] UgiMetrics, User and group related metrics
2026-05-07 14:48:09.264+08:00 DEBUG (main|1) [SecurityUtil.setTokenServiceUseIp():136] Setting hadoop.security.token.service.use_ip to true
2026-05-07 14:48:09.325+08:00 DEBUG (main|1) [Groups.getUserToGroupsMappingService():476]  Creating new Groups object
2026-05-07 14:48:09.327+08:00 DEBUG (main|1) [NativeCodeLoader.<clinit>():44] Trying to load the custom-built native-hadoop library...
2026-05-07 14:48:09.328+08:00 DEBUG (main|1) [NativeCodeLoader.<clinit>():53] Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError: no hadoop in java.library.path: /usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2026-05-07 14:48:09.328+08:00 DEBUG (main|1) [NativeCodeLoader.<clinit>():54] java.library.path=/usr/java/packages/lib:/usr/lib/x86_64-linux-gnu/jni:/lib/x86_64-linux-gnu:/usr/lib/x86_64-linux-gnu:/usr/lib/jni:/lib:/usr/lib
2026-05-07 14:48:09.328+08:00 WARN (main|1) [NativeCodeLoader.<clinit>():60] Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2026-05-07 14:48:09.329+08:00 DEBUG (main|1) [JniBasedUnixGroupsMappingWithFallback.<init>():42] Falling back to shell based
2026-05-07 14:48:09.330+08:00 DEBUG (main|1) [JniBasedUnixGroupsMappingWithFallback.<init>():46] Group mapping impl=org.apache.hadoop.security.ShellBasedUnixGroupsMapping
2026-05-07 14:48:09.349+08:00 DEBUG (main|1) [Groups.<init>():150] Group mapping impl=org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback; cacheTimeout=300000; warningDeltaMs=5000
2026-05-07 14:48:09.363+08:00 DEBUG (main|1) [UserGroupInformation$HadoopLoginModule.login():244] Hadoop login
2026-05-07 14:48:09.364+08:00 DEBUG (main|1) [UserGroupInformation$HadoopLoginModule.commit():190] hadoop login commit
2026-05-07 14:48:09.365+08:00 DEBUG (main|1) [UserGroupInformation$HadoopLoginModule.commit():216] Using user: "root" with name: root
2026-05-07 14:48:09.365+08:00 DEBUG (main|1) [UserGroupInformation$HadoopLoginModule.commit():228] User entry: "root"
2026-05-07 14:48:09.365+08:00 DEBUG (main|1) [UserGroupInformation.createLoginUser():799] UGI loginUser: root (auth:SIMPLE)
2026-05-07 14:48:09.366+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] viewfs:// = class org.apache.hadoop.fs.viewfs.ViewFileSystem from /opt/starrocks/fe/lib/hadoop-common-3.4.3.jar
2026-05-07 14:48:09.369+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] har:// = class org.apache.hadoop.fs.HarFileSystem from /opt/starrocks/fe/lib/hadoop-common-3.4.3.jar
2026-05-07 14:48:09.371+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] http:// = class org.apache.hadoop.fs.http.HttpFileSystem from /opt/starrocks/fe/lib/hadoop-common-3.4.3.jar
2026-05-07 14:48:09.372+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] https:// = class org.apache.hadoop.fs.http.HttpsFileSystem from /opt/starrocks/fe/lib/hadoop-common-3.4.3.jar
2026-05-07 14:48:09.468+08:00 DEBUG (main|1) [AbstractBackend.log():101] GHFS version: 3.0.13
2026-05-07 14:48:09.543+08:00 DEBUG (main|1) [FileSystem.loadFileSystems():3656] gs:// = class com.google.cloud.hadoop.fs.gcs.GoogleHadoopFileSystem from /opt/starrocks/fe/lib/gcs-connector-3.0.13-shaded.jar
2026-05-07 14:48:09.543+08:00 DEBUG (main|1) [FileSystem.getFileSystemClass():3692] Looking for FS supporting abfss
2026-05-07 14:48:09.544+08:00 DEBUG (main|1) [FileSystem.getFileSystemClass():3696] looking for configuration option fs.abfss.impl
2026-05-07 14:48:09.571+08:00 DEBUG (main|1) [FileSystem.getFileSystemClass():3706] Filesystem abfss defined in configuration option
2026-05-07 14:48:09.571+08:00 DEBUG (main|1) [FileSystem.getFileSystemClass():3713] FS for abfss is class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem
2026-05-07 14:48:09.572+08:00 DEBUG (main|1) [FileSystem.getFileSystemClass():3716] Jar location for class org.apache.hadoop.fs.azurebfs.SecureAzureBlobFileSystem : /opt/starrocks/fe/lib/hadoop-azure-3.4.3.jar
2026-05-07 14:48:09.577+08:00 DEBUG (main|1) [AzureBlobFileSystem.initialize():188] Initializing AzureBlobFileSystem for <abfss://drstarrockstest@ststarrockssvcdevccwus01.dfs.core.windows.net/shared_data/3c76279d-4fd6-40aa-96b5-717032118104/meta/image/automated_cluster_snapshot_1778135089752>
2026-05-07 14:48:09.591+08:00 DEBUG (main|1) [DataBlocks.createFactory():132] Creating DataFactory of type : bytebuffer
2026-05-07 14:48:09.605+08:00 DEBUG (main|1) [AzureBlobFileSystemStore.getAbfsServiceTypeFromUrl():1792] Falling back to default service type DFS
2026-
@Rocky
r
Hi there! Rocky here. This is a classic "Disaster Recovery (DR) race condition" often seen in shared-data mode when the FE is trying to pull metadata from object storage (ADLS2) while the Kubernetes environment is still trying to decide if the FE is "ready." The fact that your 
fe.log
shows the restore process starting (identifying the 
adls2://
path) but then crashing right after the Hadoop Azure (ABFS) initialization (
Falling back to default service type DFS
) is the smoking gun. Here is what's happening and how to fix it: 1. The "Default Template" vs. Operator Injection The Operator (v1.10+) has its own logic for DR. When 
disasterRecovery.enabled
is 
true
, the Operator injects its own configuration. * The Conflict: If you are manually mounting a ConfigMap to 
/opt/starrocks/fe/conf/cluster_snapshot.yaml
without using 
subPath
, you might be overwriting the directory or conflicting with the Operator's own volume mounts. * The Log Evidence: Your 
fe.log
shows a valid ADLS2 path. This means the FE is actually finding your configuration (likely from the environment variable or the Operator's mount), but the file you are seeing via 
cat
inside the pod might be the wrong one or a shadowed mount. 2. Why does it crash after 
Falling back to default service type DFS
? This is the specific point where the FE attempts to authenticate with ADLS2 to begin the download. * Missing Credentials: In shared-data mode, even for a restore, the FE needs the 
azure_adls2_shared_key
or SAS token. If these are only in your 
custom-values-dev.yaml
but not correctly projected into the FE's environment or 
fe.conf
during the restore phase, the ABFS driver will fail. * JVM Crash/OOM: Downloading a large metadata snapshot into 
meta/image
is memory-intensive. If the FE pod is restricted to 4GB or less, it might hit a Cgroup limit and be killed by the OOMKiller right as it opens the network buffers for the download. * Recommendation: Check 
kubectl describe pod
for 
Reason: OOMKilled
and check if your 
fe.conf
(or the 
cluster_snapshot.yaml
) explicitly includes the 
storage_volume
credentials for the restore. 3. The "Success on Upgrade" Mystery * On Install: The FE metadata PVC is empty. The Operator sees 
phase: doing
and adds the 
--cluster_snapshot
flag. The FE starts the restore, takes too long (or crashes), and the pod restarts. * On Upgrade: If a previous attempt partially succeeded in downloading the metadata, the PVC is no longer empty. On the next start, the FE might find the metadata on disk and skip the "restore" phase entirely, transitioning to a normal start. * Important: For a clean DR, you must delete the FE metadata PVCs before a fresh install. 4. Fix the Startup Script Timeout The message 
FE service ... 9030 is not alive yet! Timed out
comes from the 
fe_entrypoint.sh
script. * In the Helm chart, this script waits for the FE to start. If the FE is busy downloading a 1GB+ snapshot from ADLS2, port 9030 won't open. * If the script's internal timeout (often 60-120s) is shorter than the download time, the script will exit, causing the container to restart. Recommended SOS Checklist: 1. Use `subPath`: Ensure your ConfigMap mount is safe:
Copy code
yaml
   starrocksFESpec:
     configMaps:
       • name: your-dr-cm
         mountPath: /opt/starrocks/fe/conf/cluster_snapshot.yaml
         subPath: cluster_snapshot.yaml
2. Increase FE Resources: Give the FE at least 8Gi of memory and 4 CPUs for the duration of the DR restore to handle the ABFS throughput. 3. Verify 
fe.conf
Credentials: Ensure the ADLS2 credentials (
azure_adls2_endpoint
, 
azure_adls2_shared_key
) are present in the 
fe.conf
inside the pod. Without these, the 
abfss://
driver cannot authenticate. 4. Clean PVCs: If you want a fresh restore, 
kubectl delete pvc -l <http://app.kubernetes.io/name=starrocks-fe|app.kubernetes.io/name=starrocks-fe>
before the install.
5. Check StarRocks Version: Ensure you are using v3.5.0+ for the most stable ADLS2 DR support. Minimum Versions: Your Operator 
v1.11.2
is perfectly fine. The issue is almost certainly the ADLS2 credential handshake or the entrypoint script timing out during the download. References • deployment/shared_data/azure.md • administration/cluster_snapshot.md • administration/management/FE_configuration.md
Open in Slack
PreviousNext
Powered by