Hi Team, @Rocky, I have deployed a StarRocks cluster in Shared Data mode on AKS, with ADLS Gen2 configured as the storage backend. However, my FE pods are repeatedly restarting. When I checked the pod logs, I found the following error/message:

2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92647|101362) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92649|101365) [FrontendServiceImpl.loadTxnRollback():1446] failed to rollback txn 188502827: transaction not found: 188502827
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92647|101362) [TIOStreamTransport.close():153] Error closing output stream.
java.net.SocketException: Socket closed
	at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?]
	at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92649|101365) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 WARN (thrift-server-pool-92649|101365) [TIOStreamTransport.close():153] Error closing output stream.
java.net.SocketException: Socket closed
	at sun.nio.ch.NioSocketImpl.ensureOpenAndConnected(NioSocketImpl.java:165) ~[?:?]
	at sun.nio.ch.NioSocketImpl.beginWrite(NioSocketImpl.java:371) ~[?:?]
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:416) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at java.io.FilterOutputStream.close(FilterOutputStream.java:182) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:151) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.transport.TSocket.close(TSocket.java:238) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:328) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
2026-10-05 09:54:37.098+08:00 ERROR (thrift-server-pool-92650|101366) [SRTThreadPoolServer$WorkerProcess.run():321] Thrift Error occurred during processing of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Broken pipe
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:206) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:64) ~[libthrift-0.20.0.jar:0.20.0]
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40) ~[libthrift-0.20.0.jar:0.20.0]
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:313) ~[starrocks-fe.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	at java.lang.Thread.run(Thread.java:840) ~[?:?]
Caused by: java.net.SocketException: Broken pipe
	at sun.nio.ch.NioSocketImpl.implWrite(NioSocketImpl.java:425) ~[?:?]
	at sun.nio.ch.NioSocketImpl.write(NioSocketImpl.java:445) ~[?:?]
	at sun.nio.ch.NioSocketImpl$2.write(NioSocketImpl.java:831) ~[?:?]
	at java.net.Socket$SocketOutputStream.write(Socket.java:1035) ~[?:?]
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:81) ~[?:?]
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:142) ~[?:?]
	at org.apache.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:201) ~[libthrift-0.20.0.jar:0.20.0]
	... 6 more
2026-10-05 09:54:37.098+08:00 INFO (StarMgrMetaSyncer|51) [StarMgrMetaSyncer.dropTabletAndDeleteShard():277] Failed to get delete tablet response from node:104228115, error: A error occurred: errorCode=2001 errorMessage:[10.244.23.78:8060]missing tablet_ids
2026-10-05 09:54:37.098+08:00 WARN (StarMgrMetaSyncer|51) [StarMgrMetaSyncer.dropTabletAndDeleteShard():174] Failed to batch-resolve shard owners for 0 shards, falling back

The configuration for the FE deployment is as follows:

replicas: 5   # Single FE replica for development; increase for HA in production
    startupProbeFailureSeconds: 300   # Allow up to 5 minutes for FE to start up
    livenessProbeFailureSeconds: 60   # 15s default is too tight for a 16Gi-heap JVM under GC/leader load; allow 60s before restart
    readinessProbeFailureSeconds: 30  # Give a struggling FE more time before being pulled from LB rotation, without flapping

# FE configuration file (fe.conf).
    # Configures ports, JVM options, logging, load settings,
    # and shared-data storage using Azure ADLS2.
    config: |
      LOG_DIR = ${STARROCKS_HOME}/log
      DATE = "$(date +%Y%m%d-%H%M%S)"
      JAVA_OPTS="-Dlog4j2.formatMsgNoLookups=true -Xmx16384m -XX:+UseG1GC -Xlog:gc*:${LOG_DIR}/fe.gc.log.$DATE:time"

      # Network ports
      http_port = 8030
      rpc_port = 9020
      query_port = 9030
      edit_log_port = 9010

      # MySQL-compatible NIO service
      mysql_service_nio_enabled = true
      jdbc_connection_pool_size = 900

      # Logging: DEBUG level, keep 5 rolled files, auto-delete after 3 days
      sys_log_level = INFO
      sys_log_delete_age = 3d
      sys_log_roll_num = 3

      audit_log_delete_age = 3d
      audit_log_roll_num = 3

      dump_log_delete_age = 3d
      dump_log_roll_num = 3

      internal_log_delete_age = 3d
      internal_log_roll_num = 3

      big_query_log_delete_age = 3d
      big_query_log_roll_num = 3

      profile_log_delete_age = 1d
      profile_log_roll_num = 5

      # Graceful shutdown: wait up to 25 seconds before forcing exit
      min_graceful_exit_time_second = 25

      # Load profile collection for query diagnostics
      enable_load_profile = true
      load_profile_collect_interval_second = 30
      stream_load_profile_collect_threshold_second = 30

      # Trash retention for dropped catalogs: ~30 days (in seconds)
      catalog_trash_expire_second = 2592000

      # Enable loading storage volume config from fe.conf
      enable_load_volume_from_conf = true

      # Collect query detail info for observability
      enable_collect_query_detail_info = true

      # Maximum batch size for stream load (in MB)
      max_stream_load_batch_size_mb = 20480
      max_running_txn_num_per_db = 3000

      # Timeout for tablet deletion operations (in seconds)
      tablet_delete_timeout_second = 10

      # Write a cluster snapshot to ADLS2 every 5 minutes for disaster recovery
      automated_cluster_snapshot_interval_seconds = 300

      # Shared-data mode: data stored in object storage, not local BE disks
      run_mode = shared_data
      cloud_native_meta_port = 6090
      cloud_native_storage_type = ADLS2

      # ADLS2 credentials injected via environment variables from Kubernetes secret
      azure_adls2_path = ${AZURE_ADLS2_PATH}
      azure_adls2_endpoint = ${AZURE_ADLS2_ENDPOINT}
      azure_adls2_shared_key = ${AZURE_ADLS2_SHARED_KEY}

      # SSL — enables HTTPS on the FE HTTP port (8030).
      # Keystore file is mounted from the starrocks-ssl-keystore secret.
      ssl_keystore_location = /opt/starrocks/fe/conf/keystore.jks
      ssl_keystore_password = ${SSL_STORE_PASS}
      ssl_key_password = ${SSL_KEY_PASS}

@Rocky hello

@Rocky how to check if statictics collection is stuck and is in queue? in shared cluster

hey @Rocky qq - any recommended tool to implement automatic data quality checks to proactively detect issues such as missing or stale data? i.e like anomaly detection

@Rocky there should be a parameter (variable) in starrocks shared cluster which tells analyze job to trigger external column statistics update if update_time > for instance 7 days and which one is that

@Kevin Zhang please don't spam other users in slack, it's incredibly rude.

@Rocky using the merge commit with stream load works well but I do lose fidelity on tracking, I noticed PROFILE_ID in

information_schema.loads

, what is that? I'm asking because I'm seeing HTTP timeouts on loads that are successful and I don't want to retry

@Rocky - starrocks 3.5.20 - shared data mode. When restarting one of the nodes, I am getting [DefaultCoordinator.dealStatusToTryRetry():927] query af96ef80-c0f9-11f1-82b7-5e70bbafe142 failed: starlet err [RequestID=18DBBB78FACF058E][StatusCode=404]Get object s3://starrocks/a7f0ba56-f960-4602-bb98-e7be09c1fb37/db10029/10057/10059/meta/0000000000000000_000000000005F586.meta error: The specified key does not exist. How critical is it and will starrocks still be able to launch with such errors?

@Rocky We are operating Starrocks on AWS EKS, we are now investigating machine type options with Graviton. Based on current knowledge and experience, what machine type is recommended and why?

The documentation says "StarRocks supports deployments on Red Hat Enterprise Linux 7.9, CentOS Linux 7.9 or Ubuntu Linux 22.04." Do these need to be matched exactly or ar they only minimum requirements? For example would Ubuntu 24.04 be expected to work?

@Rocky I’m standing up Fluss-Paimon stack with StarRocks as query engine. Fluss 1.0.0 was recently released. Are there plans to bump the Fluss version from 0.9.1-incubating to 1.0.0? Or should I open an issue - Enhancement or Feature Request?

@Rocky I am inserting data into a PK table from a duplicate key table in a shared-data architecture, does starting another insert as soon as the first insert is complete cause instability? Is there a best practice for this type of batch ETL?

Hi @Rocky, for some reason in my shared data kubernetes deployment even though I specify requests as 20GB ram per CN, after the compute nodes come up, I only see 16.2GB as MemLimit when I run

SHOW COMPUTE NODES;

any ideas why?

@Rocky can I create materialized views into an external catalog?

@Rocky What improvements are there going from 4.1.4 to 4.1.6 as of now 4.1.4 and 4.1.6 both not visible in github release is there any other place i can see cahnge log

This message was deleted.

@Rocky how to run a query on http 8030 port from python?

Is there any ETA on when Hybrid row-column will come out of beta ?

In spirit of ETA questions is there any ETA for going GA for native inverted index (

imp_lib=builtin

) for full-text search in shared-data mode

@Rocky is starrocks version 4.1.6 last patch for 4.1.* version? when is it gonna be official released?

@Rocky We run StarRocks 4.1 shared-data, 3 CN and 3 FE. When we run a large INSERT INTO ... SELECT FROM FILES() (parquet on S3), CN network in/out grows a lot (~500–600 MB/s per CN). CPU stays around 25% and memory looks fine. At the same time our interactive SELECTs on other tables get slow. Some hit a 10s deadline and cancel with ScanRows=0 / ErrorCode CANCELLED — they never start a storage scan. After the load finishes, the same SQL is fast again (hundreds of ms). Loads and queries share the same CNs, same default_warehouse and default_wg. No separate CN group. What do you recommend in 4.1 shared-data to keep large FILES loads from stalling interactive queries? Separate warehouse + CN group? Resource groups on the same CNs? Something else? Any limits or gotchas on a 3-CN cluster?

Is there a BE memory leak issue with Starrocks 3.5 version? Our deployed SR3.5 version has consistently high memory, which can only be resolved after restarting

@Rocky I'm using starrocks 4.0.8 shared architecutre, I'm getting compute node already exists for cn-4, even though I did drop compute node and scale up pod still getting same. How can I fixt his? [Fri Oct 9 123437 IST 2026] Process conf file cn.conf ... [Fri Oct 9 123437 IST 2026] Add myself (starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local:9050) into FE ... ERROR 1064 (HY000) at line 1: Compute node already exists with same host starrocks-cn-4.starrocks-cn-search.starrocks-shared.svc.cluster.local and port 9050

@Rocky why dont have scheduled delete in starrocks ?

This message was deleted.

This message was deleted.

@Rocky hello. I want to download fresh version. But it empty https://www.starrocks.io/download/community/index.html

@Rocky who is the repo maintainer to @ when assigning an issue

@Rocky we try to import data from starrocks to powerbi via mysql odbc driver and I see when it is on the finishline it starts the query on starrocks side all over again and powerbi just waiting for the import ://

@Rocky Running StarRocks 3.5.20 shared-data with 3 FEs on JDK 17.0.20, -Xms6g -Xmx6g -XX:MaxDirectMemorySize=3g, MALLOC_ARENA_MAX=2, and Arrow Flight SQL in proxy mode. FE processes get kernel OOM-killed at about 11 GiB RSS. On one FE, 15 minutes after restart: RSS 11.3 GiB, NMT committed 7.8 GiB (heap 6.0, Other 1.0). jcmd <pid> System.trim_native_heap freed 5.4 GiB (RSS 11.7 GB to 6.2 GB), so most of the growth is freed native memory that glibc keeps, probably from Arrow/Netty buffers during large Flight result sets. 1. Is FE native memory retention under Arrow Flight a known issue? Any GitHub issue or fix? 2. Does StarRocks recommend an allocator setting for the FE (jemalloc, MALLOC_TRIM_THRESHOLD_, or -XX:TrimNativeHeapInterval)? 3. Is there any backpressure or memory limit for Arrow Flight proxy mode on the FE?