http://starrocks.io logo
Join Slack
Powered by
Hi Team, I have deployed a StarRocks CN cluster on...
# questions-and-troubleshooting
a
Hi Team, I have deployed a StarRocks CN cluster on AKS using a Helm chart. It is a shared-data cluster, and the data is synced to ADLS. I used a custom values file for the deployment (attached below). However, my FE pods and CN nodes keep restarting repeatedly. Has anyone faced this issue before or found a solution? Any help would be greatly appreciated. error getting:
Copy code
kubectl logs kube-starrocks-cn-4 -n starrocks --previous --tail=500
[Mon Mar  2 14:10:20 CST 2026] Process conf file cn.conf ...
[Mon Mar  2 14:10:20 CST 2026] Add myself (kube-starrocks-cn-4.kube-starrocks-cn-search.starrocks.svc.cluster.local:9050) into FE ...
ERROR 1064 (HY000) at line 1: Compute node already exists with same host kube-starrocks-cn-4.kube-starrocks-cn-search.starrocks.svc.cluster.local and port 9050
[Mon Mar  2 14:10:20 CST 2026] run start_cn.sh
/opt/starrocks/cn_entrypoint.sh: line 165:    27 Segmentation fault      (core dumped) $STARROCKS_HOME/bin/start_cn.sh $addition_args
Copy code
kubectl exec -it kube-starrocks-cn-2 -n starrocks -- /bin/sh
[warn] Error from accept() call: Invalid argument
[warn] Error from accept() call: Invalid argument
[warn] Error from accept() call: Invalid argument
start time: Mon Mar  2 14:09:40 CST 2026, server uptime:  14:09:40 up 10 days, 19:38,  0 users,  load average: 1.79, 5.10, 7.29
Run with JEMALLOC_CONF: 'percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000,metadata_thp:auto,background_thread:true,prof:true,prof_active:false'
4.0.6 RELEASE (build 45fdede distro ubuntu arch x86_64)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000, plan_node_id:-1
*** Aborted at 1772432489 (unix time) try "date -d @1772432489" if you are using GNU date ***
PC: @          0x8c9eb5b std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
*** SIGSEGV (@0x0) received by PID 27 (TID 0x7f0c451ff640) LWP(8862) from PID 0; stack trace: ***
    @     0x7f0d9f222ee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11f1dba8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f0d9f1cb520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0x8c9eb5b std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @         0x1460d754 Azure::Storage::Files::DataLake::_detail::FileSystemClient::ListPaths(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Files::DataLake::_detail::FileSystemClient::ListFileSystemPathsOptions const&, Azure::Core::Context .
    @         0x145be80c Azure::Storage::Files::DataLake::DataLakeDirectoryClient::ListPaths(bool, Azure::Storage::Files::DataLake::ListPathsOptions const&, Azure::Core::Context const&) const
    @          0xedba5cd staros::starlet::fslib::ADLS2FileSystem::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xedd709e staros::starlet::fslib::CacheFileSystemImpl::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xedd4716 staros::starlet::fslib::CacheFileSystem::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xccae039 starrocks::StarletFileSystem::iterate_dir(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (std::basic_string_view<char, std::char_traits<char> >)> const&)
    @          0xdc2c7e7 starrocks::lake::delete_tablets_impl(starrocks::lake::TabletManager*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<long, std::allocator<long> > const&)
    @          0xdc2e6e4 starrocks::lake::delete_tablets(starrocks::lake::TabletManager*, starrocks::DeleteTabletRequest const&, starrocks::DeleteTabletResponse*)
    @          0xe1a6d9c std::_Function_handler<void (), starrocks::LakeServiceImpl::delete_tablet(google::protobuf::RpcController*, starrocks::DeleteTabletRequest const*, starrocks::DeleteTabletResponse*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const&+
    @          0xe3ab83e starrocks::ThreadPool::dispatch_thread()
    @          0xe3a23e5 starrocks::Thread::supervise_thread(void*)
    @     0x7f0d9f21dac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f0d9f2af8c0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268bf)
[1772432490.394][thread: 139690676057664] je_mallctl execute purge success
[1772432490.394][thread: 139690676057664] je_mallctl execute dontdump success
start time: Mon Mar  2 14:21:41 CST 2026, server uptime:  14:21:41 up 10 days, 19:50,  0 users,  load average: 49.58, 13.31, 8.14
Run with JEMALLOC_CONF: 'percpu_arena:percpu,oversize_threshold:0,muzzy_decay_ms:5000,dirty_decay_ms:5000,metadata_thp:auto,background_thread:true,prof:true,prof_active:false'
4.0.6 RELEASE (build 45fdede distro ubuntu arch x86_64)
query_id:00000000-0000-0000-0000-000000000000, fragment_instance:00000000-0000-0000-0000-000000000000, plan_node_id:-1
*** Aborted at 1772433787 (unix time) try "date -d @1772433787" if you are using GNU date ***
PC: @          0x8c9eb5b std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
*** SIGSEGV (@0x0) received by PID 27 (TID 0x7f5135de9640) LWP(7683) from PID 0; stack trace: ***
    @     0x7f539ff2aee8 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x99ee7)
    @         0x11f1dba8 google::(anonymous namespace)::FailureSignalHandler(int, siginfo_t*, void*)
    @     0x7f539fed3520 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x4251f)
    @          0x8c9eb5b std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >::_M_assign(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)
    @         0x1460d754 Azure::Storage::Files::DataLake::_detail::FileSystemClient::ListPaths(Azure::Core::Http::_internal::HttpPipeline&, Azure::Core::Url const&, Azure::Storage::Files::DataLake::_detail::FileSystemClient::ListFileSystemPathsOptions const&, Azure::Core::Context 

    @         0x145be80c Azure::Storage::Files::DataLake::DataLakeDirectoryClient::ListPaths(bool, Azure::Storage::Files::DataLake::ListPathsOptions const&, Azure::Core::Context const&) const
    @          0xedba5cd staros::starlet::fslib::ADLS2FileSystem::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xedd709e staros::starlet::fslib::CacheFileSystemImpl::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xedd4716 staros::starlet::fslib::CacheFileSystem::list_dir(std::basic_string_view<char, std::char_traits<char> >, bool, std::function<bool (staros::starlet::fslib::EntryStat)>, std::basic_string_view<char, std::char_traits<char> >)
    @          0xccae039 starrocks::StarletFileSystem::iterate_dir(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::function<bool (std::basic_string_view<char, std::char_traits<char> >)> const&)
    @          0xdc2c7e7 starrocks::lake::delete_tablets_impl(starrocks::lake::TabletManager*, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::vector<long, std::allocator<long> > const&)
    @          0xdc2e6e4 starrocks::lake::delete_tablets(starrocks::lake::TabletManager*, starrocks::DeleteTabletRequest const&, starrocks::DeleteTabletResponse*)
    @          0xe1a6d9c std::_Function_handler<void (), starrocks::LakeServiceImpl::delete_tablet(google::protobuf::RpcController*, starrocks::DeleteTabletRequest const*, starrocks::DeleteTabletResponse*, google::protobuf::Closure*)::{lambda()#1}>::_M_invoke(std::_Any_data const& 
    @          0xe3ab83e starrocks::ThreadPool::dispatch_thread()
    @          0xe3a23e5 starrocks::Thread::supervise_thread(void*)
    @     0x7f539ff25ac3 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x94ac2)
    @     0x7f539ffb78c0 (/usr/lib/x86_64-linux-gnu/libc.so.6+0x1268bf)
[1772433788.303][thread: 139986772858432] je_mallctl execute purge success
[1772433788.303][thread: 139986772858432] je_mallctl execute dontdump success
custom-values-prod.yaml
k
It crashed during clean the storage associated with the tablet, 1) find the tablet id from FE log side, it should be something from deleteTablets rpc error. 2), find the storage path for tablet, and list the files under the path, check if there is any abnormal under the path.
a
@Kevin Cai I couldn’t find the table in the logs. It looks like a different one.
Copy code
at java.base/java.lang.Thread.run(Thread.java:840)

Feb 25, 2026 9:16:16 PM com.baidu.jprotobuf.pbrpc.transport.RpcChannelFutureListener operationComplete
WARNING: build channel:[id: 0x9c6a1802, L:null ! R:/10.244.32.30:8060] failed
Feb 25, 2026 9:16:16 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory wrap
SEVERE: failed to get result from stp
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.244.32.30:8060
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:684)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

Feb 25, 2026 9:16:16 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPool getChannel
SEVERE: Unable to validate object
java.util.NoSuchElementException: Unable to validate object
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:506)
	at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:363)
	at com.baidu.jprotobuf.pbrpc.transport.ChannelPool.getChannel(ChannelPool.java:80)
	at com.baidu.jprotobuf.pbrpc.transport.RpcChannel.getConnection(RpcChannel.java:73)
	at com.baidu.jprotobuf.pbrpc.client.ProtobufRpcProxy.invoke(ProtobufRpcProxy.java:499)
	at jdk.proxy2/jdk.proxy2.$Proxy35.deleteTablet(Unknown Source)
	at com.starrocks.rpc.LakeServiceWithMetrics.deleteTablet(LakeServiceWithMetrics.java:84)
	at com.starrocks.lake.StarMgrMetaSyncer.dropTabletAndDeleteShard(StarMgrMetaSyncer.java:167)
	at com.starrocks.lake.StarMgrMetaSyncer.cleanOneGroup(StarMgrMetaSyncer.java:317)
	at com.starrocks.lake.StarMgrMetaSyncer.deleteUnusedShardAndShardGroup(StarMgrMetaSyncer.java:285)
	at com.starrocks.lake.StarMgrMetaSyncer.runAfterCatalogReady(StarMgrMetaSyncer.java:544)
	at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)
	at com.starrocks.common.util.Daemon.run(Daemon.java:98)

Feb 25, 2026 11:40:24 PM com.baidu.jprotobuf.pbrpc.transport.RpcChannelFutureListener operationComplete
WARNING: build channel:[id: 0x6bf70c92, L:null ! R:/10.244.32.30:8060] failed
Feb 25, 2026 11:40:24 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory wrap
SEVERE: failed to get result from stp
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.244.32.30:8060
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:684)
	at java.base/sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:946)
	at io.netty.channel.socket.nio.NioSocketChannel.doFinishConnect(NioSocketChannel.java:336)
	at io.netty.channel.nio.AbstractNioChannel$AbstractNioUnsafe.finishConnect(AbstractNioChannel.java:339)
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:784)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:732)
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:658)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:998)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:840)

Feb 25, 2026 11:40:24 PM com.baidu.jprotobuf.pbrpc.transport.ChannelPoolObjectFactory wrap
SEVERE: failed to get result from stp
io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /10.244.32.30:8060
This is the Error code I'm getting
Copy code
State:          Running
      Started:      Mon, 02 Mar 2026 18:08:08 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    139
      Started:      Mon, 02 Mar 2026 18:03:43 +0530
      Finished:     Mon, 02 Mar 2026 18:08:07 +0530
🚨 SOS – CN Pod Crashing 🚨 Hi team, We’re experiencing repeated crashes on the CN pod, and it’s affecting service stability. If anyone has faced this issue before or has a possible solution, please help
k
is there large memory allocation warn log found in 
cn.WARN
?
one possible workaround is to 
ADMIN SET FRONTEND CONFIG('meta_sync_force_delete_shard_meta' = 'true');
skipping remote file cleanup on azure, so the cluster can be recovered. turn it for a few hours and then turn it off. However, the real root cause is still unknown, may experience the same issue in future.
Open in Slack
PreviousNext
Powered by