Hi Team, @Rocky, I have deployed a StarRocks cluster in Shared Data mode on AKS, with ADLS Gen2 configured as the storage backend. However, my FE pods are repeatedly restarting. When I checked the pod logs, I found the following error/message:

2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92647|101362) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92649|101365) [FrontendServiceImpl.loadTxnRollback():1446] failed to rollback txn 188502827: transaction not found: 188502827
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92647|101362) [TIOStreamTransport.close():153] Error closing output stream.
java.net.SocketException: Socket closed
	at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?]
	at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92649|101365) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92649|101365) [TIOStreamTransport.close():153] Error closing output stream.
java.net.SocketException: Socket closed
	at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?]
	at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92650|101366) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 INFO (StarMgrMetaSyncer|51) [StarMgrMetaSyncer.dropTabletAndDeleteShard():277] Failed to get delete tablet response from node:104228115, error: A error occurred: errorCode=2001 errorMessage:[10.244.23.78:8060]missing tablet_ids
2026-10-05 09:54:37.098+08:00 WARN (StarMgrMetaSyncer|51) [StarMgrMetaSyncer.dropTabletAndDeleteShard():174] Failed to batch-resolve shard owners for 0 shards, falling back

The configuration for the FE deployment is as follows:

replicas: 5   # Single FE replica for development; increase for HA in production
    startupProbeFailureSeconds: 300   # Allow up to 5 minutes for FE to start up
    livenessProbeFailureSeconds: 60   # 15s default is too tight for a 16Gi-heap JVM under GC/leader load; allow 60s before restart
    readinessProbeFailureSeconds: 30  # Give a struggling FE more time before being pulled from LB rotation, without flapping

# FE configuration file (fe.conf).
    # Configures ports, JVM options, logging, load settings,
    # and shared-data storage using Azure ADLS2.
    config: |
      LOG_DIR = ${STARROCKS_HOME}/log
      DATE = "$(date +%Y%m%d-%H%M%S)"
      JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx16384m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"

      # Network ports
      http_port = 8030
      rpc_port = 9020
      query_port = 9030
      edit_log_port = 9010

      # MySQL-compatible NIO service
      mysql_service_nio_enabled = true
      jdbc_connection_pool_size = 900

      # Logging: DEBUG level, keep 5 rolled files, auto-delete after 3 days
      sys_log_level = INFO
      sys_log_delete_age = 3d
      sys_log_roll_num = 3

      audit_log_delete_age = 3d
      audit_log_roll_num = 3

      dump_log_delete_age = 3d
      dump_log_roll_num = 3

      internal_log_delete_age = 3d
      internal_log_roll_num = 3

      big_query_log_delete_age = 3d
      big_query_log_roll_num = 3

      profile_log_delete_age = 1d
      profile_log_roll_num = 5

      # Graceful shutdown: wait up to 25 seconds before forcing exit
      min_graceful_exit_time_second = 25

      # Load profile collection for query diagnostics
      enable_load_profile = true
      load_profile_collect_interval_second = 30
      stream_load_profile_collect_threshold_second = 30

      # Trash retention for dropped catalogs: ~30 days (in seconds)
      catalog_trash_expire_second = 2592000

      # Enable loading storage volume config from fe.conf
      enable_load_volume_from_conf = true

      # Collect query detail info for observability
      enable_collect_query_detail_info = true

      # Maximum batch size for stream load (in MB)
      max_stream_load_batch_size_mb = 20480
      max_running_txn_num_per_db = 3000

      # Timeout for tablet deletion operations (in seconds)
      tablet_delete_timeout_second = 10

      # Write a cluster snapshot to ADLS2 every 5 minutes for disaster recovery
      automated_cluster_snapshot_interval_seconds = 300

      # Shared-data mode: data stored in object storage, not local BE disks
      run_mode = shared_data
      cloud_native_meta_port = 6090
      cloud_native_storage_type = ADLS2

      # ADLS2 credentials injected via environment variables from Kubernetes secret
      azure_adls2_path = ${AZURE_ADLS2_PATH}
      azure_adls2_endpoint = ${AZURE_ADLS2_ENDPOINT}
      azure_adls2_shared_key = ${AZURE_ADLS2_SHARED_KEY}

      # SSL — enables HTTPS on the FE HTTP port (8030).
      # Keystore file is mounted from the starrocks-ssl-keystore secret.
      ssl_keystore_location = /opt/starrocks/fe/conf/keystore.jks
      ssl_keystore_password = ${SSL_STORE_PASS}
      ssl_key_password = ${SSL_KEY_PASS}

Your StarRocks Frontend (FE) pods are likely restarting because they are failing their Kubernetes liveness/readiness probes or encountering a quorum failure. The "Broken pipe" and "missing tablet_ids" errors are often symptoms of the FE process shutting down or communication being interrupted, rather than the root cause of the restart itself. Based on your configuration and logs, here are the most likely causes and recommended fixes: 1. Version Compatibility (Critical) The parameters

cloud_native_storage_type = ADLS2

and

azure_adls2_path

were introduced in StarRocks v3.4.1. * If you are on a version earlier than 3.4.1: The FE will not recognize the

ADLS2

storage type in

fe.conf

. In older versions, you must use

AZBLOB

(which works for ADLS Gen2 using the blob endpoint) or upgrade to v3.4.1+. * Action: Verify your StarRocks version. If

< 3.4.1

, change your config to:

properties
    cloud_native_storage_type = AZBLOB
    azure_blob_path = <container>/<path>
    azure_blob_endpoint = https://<account>.<http://blob.core.windows.net|blob.core.windows.net>
    azure_blob_shared_key = <key>

2. Quorum and Network Issues You are running 5 FE replicas. In StarRocks, FEs use BDBJE for metadata synchronization and require a majority (3 out of 5) to be healthy to form a quorum. * Missing `priority_networks`: In Kubernetes, Pods have multiple network interfaces. If

priority_networks

is not set in

fe.conf

, FEs might bind to the wrong IP, preventing them from communicating with each other. This causes them to fail to join the cluster and restart. * Action: Add

priority_networks

to your

fe.conf

. For AKS (standard CNI), this is typically:

properties
    priority_networks = 10.244.0.0/16  # Adjust to match your Pod CIDR

3. Memory Resource Limits (OOMKilled) You have configured

-Xmx16384m

(16Gi). * If your Kubernetes Pod resource limit is also set to 16Gi, the Pod will be OOMKilled by the kernel. The JVM needs additional memory for overhead (Metaspace, thread stacks, direct buffers). * Action: Ensure your Pod's memory limit is at least 10-20% higher than the JVM heap (e.g., set the Pod limit to

20Gi

for a

16Gi

heap). Use

kubectl describe pod <fe-pod>

to check the

Last State: Terminated

reason; if it says

OOMKilled

, this is your issue. 4. "Missing tablet_ids" and "Broken Pipe" Errors * Missing tablet_ids (Error 2001): This occurs when the

StarMgrMetaSyncer

(a background task in shared-data mode) sends an empty batch-delete request to a Backend (BE). While it looks like a severe error, it is often a side effect of metadata inconsistency or the FE being in a "shutting down" state. * Broken Pipe: This indicates a network connection was closed abruptly. Since your FEs are restarting, this usually means the FE process died while a Thrift RPC (like a heartbeat or metadata sync) was in progress. 5. ADLS Gen2 Hierarchical Namespace * Important: StarRocks shared-data mode requires the Azure Storage account to have Hierarchical Namespace (HNS) disabled if using the

AZBLOB

type. If you are using the

ADLS2

type (v3.4.1+), HNS is supported. * Action: Double-check that your

AZURE_ADLS2_PATH

environment variable does not include the protocol prefix (e.g., it should be

mycontainer/datapath

, not `adls2://mycontainer/datapath`). Recommended Troubleshooting Steps 1. Check Pod Exit Reason: Run

kubectl get pods

and check

kubectl describe pod <fe-name>

. Look for

OOMKilled

or

BackOff

. 2. Verify Quorum: Check the logs of the first FE pod for "master/leader election" messages. If it can't find other nodes, verify the

FE_SERVERS

environment variable or the

metadata_failure_recovery

status. 3. Scale Down to 1 FE: For debugging, scale your deployment to

replicas: 1

. If it stabilizes, the issue is likely network connectivity between FE replicas (Quorum/BDBJE).