faced with such a problem, I'm brewing starrocks i...
# questions-and-troubleshooting
a
faced with such a problem, I'm brewing starrocks in k8c through an operator. we launch the deployment, everything rises, we take 3 fe hearths. if the fe runs on a single node, then everything is ok. but if it is different, he cannot receive the token on port 9010/8030. and under the name, for example, fe-1 does not start. The analysis showed that the fe-service will allow traffic to pod, there after it is ready. but the pod cannot be ready because the service does not allow traffic. what to do?
k
is the istio enabled for the k8s deployment?
a
not enabled. new cluster, empty. Only starrocks
k
also the error log
device or resource busy
from pod fe-1, is suspecious.
a
yes
fe.conf from config.map
Снимок экрана 2025-12-08 в 09.36.57.png
k
the entrypoint will try to update the fe.conf and link to the final location. are you manually change it to the final location?
a
Copy code
looked at the latest logs
Снимок экрана 2025-12-08 в 09.39.48.png
if i do alterr system remove follower, fe will try to join again and the result will be the same
Снимок экрана 2025-12-08 в 09.46.12.png
Снимок экрана 2025-12-08 в 09.45.32.png
Снимок экрана 2025-12-08 в 09.47.49.png
fe leader logs
Снимок экрана 2025-12-08 в 09.48.20.png
Copy code
He tries to get a token, but since the service has not yet started, he gets a connection refused message. I checked the ports, 8030 and the others are available on the leader, but not on this follower.
k8s 1.30 chart 1.11.3 starrocks image tag 3.3-latest
k
so the blocker is, the fe-1 (follower) started and bootstrap itself, but fe-0 (the leader) still can't make connect to the fe-1, need to check why fe-0 can't connect to fe-1 while fe-1 is up and wait for the bootstrap.
a
Снимок экрана 2025-12-08 в 10.08.55.png
Copy code
It doesn't start, which means the service isn't sending traffic to it. There's no endpoint.
k
it contact fe-1 with the
fe-search
service, not this
fe-service
service.
a
Снимок экрана 2025-12-08 в 10.11.28.png
k
shall check the endpoint behind the
fe-search
a
Снимок экрана 2025-12-08 в 10.12.55.png
Copy code
Until the main service is up, it won't route traffic or open ports. I even tried deleting the probes, but the result is the same.
k
why it won't route traffic or open ports by this
fe-search
, it sets to ready immediately without checking the pod's probing.
a
Copy code
As I understand it, he doesn't consider it ready for himself until he exchanges with the leader. I'm trying to solve this problem.
Copy code
I ran nginx on the same node, and when it started, traffic started flowing. It turns out the k8s node itself doesn't open the port until the pod starts.
@Kevin Cai can you help ?
Copy code
The probes are running on the same port as the web server.
k
fe-search service doesn't do any probing before registering the pod into its endpoint. https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/pkg/k8sutils/templates/service/spec.go#L40 if the service
fe-search
in your env doesn't have fe-1 endpoint registered when the pod is up (not ready yet, just up), must be something wrong this service setup.
a
I am installed starrocks from chart
k
so it must be some other issues. not the fe-search service endpoint issue. possibly when fe-1 pod starts, manual check its connectivity inside fe-0 pod, if the connection refused to fe-1 can be reproduced consistently.
a
I checked and get connetcion refused
Снимок экрана 2025-12-08 в 15.23.23.png
k
double check that tcp/8030 is opened or not in fe-1 pod
a
the ports are specified in sts
fe1 to fe 0
fe 1 to fe 0
Снимок экрана 2025-12-08 в 15.46.33.png
fe-search have endpoint but port 9030 query mysqk
If I change or add 8030, the operator overwrites it
k
it has nothing to do with the
fe-search
configuration, as you can see, you can't even reach the fe-1 8030 by ip address, it doesn't go through the fe-search service
so 1. ensure the fe-1 tcp/8030 is listening, 2, investigate why nc to fe-1 ipaddress:8030 get refused.
a
No one is listening.
2, investigate why nc to fe-1 ipaddress:8030 get refused. becouse pod, not start. I took nginx and launched it on the same node as the fe-1, as soon as nginx started, port 8030 opened
What am I misunderstanding?
k
when fe-1 is not registered to fe-0. the log will look like fe-0
Copy code
2025-12-08 13:31:58.629Z WARN (nioEventLoopGroup-4-1|147) [MetaBaseAction.isFromValidFe():114] request is not from valid FE. client: 172.21.0.101
fe-1
Copy code
2025-12-08 13:32:38.687Z WARN (main|1) [NodeMgr.getFeNodeTypeAndNameFromHelpers():523] failed to get fe node type from helper node: 172.21.0.100:9010. response code: 400
2025-12-08 13:32:38.687Z WARN (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():372] current node is not added to the group. please add it first. sleep 5 seconds and retry, current helper nodes: [172.21.0.100:9010]
and when fe-1 is added to fe-0 via
alter system add follower ...
fe-1
Copy code
2025-12-08 13:32:43.696Z INFO (main|1) [NodeMgr.getFeNodeTypeAndNameFromHelpers():553] get fe node type FOLLOWER, name 172.21.0.101_9010_1765200762950 from 172.21.0.100:8030
2025-12-08 13:32:43.705Z INFO (main|1) [NodeMgr.getVersionFileFromHelper():701] Downloading version file from <http://172.21.0.100:8030/version>
2025-12-08 13:32:43.763Z INFO (main|1) [NodeMgr.getNewImageOnStartup():743] skip download image for /opt/starrocks/fe/meta/image, current version 0 >= version 0 from 172.21.0.100:9010
2025-12-08 13:32:43.763Z INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():499] Current run_mode is shared_nothing
2025-12-08 13:32:43.764Z INFO (main|1) [NodeMgr.getClusterIdAndRoleOnStartup():504] Got role: FOLLOWER, node name: 172.21.0.101_9010_1765200762950 and run_mode: shared_nothing
2025-12-08 13:32:43.765Z INFO (main|1) [BDBEnvironment.ensureHelperInLocal():351] start to check if local replica environment from /opt/starrocks/fe/meta/bdb contains 172.21.0.100:9010
a
I did alter system add follower, the result is the same (
k
wait. I saw your pod's ip addresses are in the range of
10.230.x.x
, and it shows the request from
10.128.0.16
in fe-0 log, which the fe-1's http request get rejected.
are the pods bind with multiple ip addresses? which ip address does the dns name for pod fe-1 resolve to?
a
Снимок экрана 2025-12-08 в 16.48.42.png
cluster pod 10.96.0.0/16 ()mask
k
why fe-0 said the request is from 10.128.0.16, is it the node ip address where the fe-1 is hosted by?
what's the cni used for the k8s cluster?
a
when it restarts, the ip changes, it is added to the database as an FQDN
k
that's the expected behavior. the fe-1 is added to fe-0 with its fqdn hostname, and then fe-0 will validate the incoming http request via the remote address of the tcp connection, and usually this should match the ip address that resolves from the fqdn for the fe-1, however, in your env, it resolves to 10.128.0.16 rather than 10.230.132.35 or whatever in 10.230.132.x, that's why fe-0 rejects the http request from fe-1, and fe-1 can't bootstrap itself from fe-0's help.
a
by default flannel
latest configuration
Снимок экрана 2025-12-08 в 17.00.08.png
k
kubectl get nodes -o wide
a
Снимок экрана 2025-12-08 в 17.03.22.png
k
that's it! 10.128.x.x is the node ip address, must be something wrong with the network setup in the k8s cluster.
a
why ?
what should be the range? I will create
Network Settings Cloud network default Cluster CIDR 10.230.0.0/16 CIDR services 10.96.0.0/16 Subnet mask of 24 nodes Max. number of nodes 128 Max. number of hearths per node 110
his network intersects with compute nodes
k
what I can tell from fe-0's log, seems underlay ip address (node network 10.128.x.x) is exposed to overlay network (pod network, 10.230.0.0/16) where when a client connection from fe-1 reaches fe-0, fe-0 doesn't see the connection from overlay (10.230.x.x, suppose to be the fe-1 pod ip address) but actually the underlay ip address 10.128.x.x which is the ip address of the node who hosts the fe-1 pod. And this also explains when fe-1 pod residents the same node as fe-0, the issue is gone, because they are on the same k8s node, no cross-node network translation, the overlay ip address is preserved.
Trouble shoot of flannel network setup (or k8s network setup) is out of my domain knowledge.
maybe you can give a try to remove the nodePort setup for the
sr-port-service
, not sure if it impacts somehow. this is just my wild guess, may not make sense at all.
a
I created the sr-port-service trying to solve the problem.
Copy code
experimented for a couple of hours.
What assortment would you recommend?
same result
Снимок экрана 2025-12-08 в 22.06.09.png
Снимок экрана 2025-12-08 в 23.27.42.png
removed readiness probe
k
looks like finally the cluster is up successfully.
a
Yes, I removed the readness probe and added it to kube-starrocks-fe-service PublishNotReadyAddresses: true then it worked
have another problem,
Copy code
A client connected via a load balancer makes a request from the Kubernetes node, not through the service....
k
so the problem is not solved yet.
a
I think I figured out what the problem is. Look at the port, it's NAT. It looks like we need different routing within the cloud.
k
should be vxlan for the overlay network in kubernetes, why it is a NAT?
a
Снимок экрана 2025-12-10 в 08.44.55.png
Copy code
The K8S load balancer has a static public IP address, and the airflow request goes over the internet. Using virtual cables, you get two computers from different networks connected via a single cable. This makes them accessible via OSI Layer 2 over MAC. I'll test this theory today.
Copy code
For some reason, a request is coming from the node's IP address?
k
yeah, that's the root cause where the problem occures
a
No, the request still arrives, what's wrong?
Copy code
2025-12-10 16:00:00.286+03:00 WARN (starrocks-mysql-nio-pool-3|190) [StmtExecutor.execute():800] execute Exception, sql SHOW COMPUTE NODES
Copy code
`org.apache.thrift.TApplicationException: Internal error processing forward`
Copy code
How is this pool assembled? Can I look at the code?
k
try to run 'show compute nodes' from leader fe, so that the request won't be forwarded to the leader, and hence won't be rejected by leader due to incoming connection remote ip address mismatch.
a
i understand it
how to fill this variable ? $(FE_SERVICE_NAME)
Copy code
We did this, and it handles single requests fine. But when we launch Airflow, we get this error. As far as I understand, some requests are coming from the Kubernetes node's IP and are being rejected.
I've contacted Cloud support, and maybe they can explain.
Copy code
requests once per minute, request with Internal error processing forward, once every three minutes.
There's no router on this network, and I think everything there is running on the second OSI level. Third, there's simply no one to route it.
@Kevin Cai HI.
Copy code
Can you tell me what problem you're seeing? Again?
Cloud support said everything's fine, check your Starrocks settings.
I've read a lot of issues and seen that detailed problems are resolved by initing the container that parses DNS names, and then configuring fe.conf based on that.
Something like this.
      - bash
      - '-c'
      - |
      set -ex
      hostname=`hostname`
      ip=`ifconfig eth0|grep inet|awk '{print $2}'`
      echo "$(sed 's/^'"$ip"'.*/'"$ip"'        '"$hostname"'.starrocks-shared-data-fe-service.***.svc.dev-blz1.hefei '"$hostname"'/g' /etc/hosts)" > /etc/hosts
      [[ $hostname =~ -([0-9]+)$ ]] || exit 1
      ordinal=${BASH_REMATCH[1]}
       if [[ $ordinal -eq 0 ]]; then
           #first pod as leader default, FQDN host type
           /opt/starrocks/fe/bin/start_fe.sh --host_type FQDN
       else
           #other pods as followers,helper is the first pod's service, FQDN host type 
           /opt/starrocks/fe/bin/start_fe.sh --helper starrocks-shared-data-fe-0.starrocks-shared-data-fe-service.***.svc.dev-blz1.***i:9010 --host_type FQDN 
       fi
k
That should not be the right thing to do.
you can try to deploy sr on EKS or GKE, it is almost no addition configuration and just work.
a
It's just that when pods register, they don't get the DNS name for Kubernetes as quickly, and this might be why the cluster isn't being formed correctly. I don't know what information it passes in the metadata when it's built. So, initcontainer seems like a good solution.
Copy code
Of course, you can try and then look at the service manifests; there might be some option that will solve everything.
@Kevin Cai Interesting point: I connect according to the instructions, create two variables, log into the StarRox console, make requests, and immediately receive a request for the node's IP. Is it possible to find out how StarRox receives them?
Copy code
It would be interesting to understand and solve the problem)
k
this is because you connect to starrocks FE follower , and the follower forward your request to the FE leader. However, FE leader checks the incoming connection IP address from the follower, and reject it because it shows the ip is not the same as the one resolved by FQDN.
the key thing is, when a tcp connection is established, it will be (local_ip, local_port, remote_ip, remote_port), FE will check the remote_ip, ensure it matches the one registered as Frontend follower. This is a IP allowlist behavior.
a
Copy code
Is there a way to specify allowed networks? By narrowing the range?
What information could I give you? To help figure this out? Maybe I should call? I could show you the configuration, and you could tell me what you think? I understand the problem is getting on my nerves. But a solution needs to be found. How many more clients like this can we have? The product is still developing.
Copy code
You can say, "I need manifests for this and that," I'll generate them and send them to your email. Take a look.
@Kevin Cai
Well, we solved the problem, installed the Calico driver, and everything worked. Your link to GKE helped, since Callico is installed by default there.
👍 1
It turns out that with Flannel you either need additional settings or some modifications to Starrocks to make it work.