Has anyone successfully implemented Scaling Down using the H StarRocks #questions-and-troubleshooting

Has anyone successfully implemented Scaling Down u...

JD Baudean

10/15/2025, 2:17 PM

Has anyone successfully implemented Scaling Down using the Horizontal Pod Autoscaler? The example Starrocks provides has it disabled. I tried adding a basic scale down section and I'm still having to manually DROP the CN from the database, which pretty much defeats the purpose of autoscaling. This is in a 3.5 shared data cluster running in AKS. https://docs.starrocks.io/docs/deployment/sr_operator/#automatic-scaling-for-cn-cluster

JD Baudean

10/17/2025, 8:20 PM

This appeared to be a bug in the operator. https://github.com/StarRocks/starrocks/issues/62915 I upgraded from 1.10.2 to 1.11.2 and that fixed it

JD Baudean

10/20/2025, 3:29 PM

Well, it works sometimes and not others. Can anyone tell me if using the HPA to scale down CNs is officially supported? It seems to affect running queries too but I'm not entirely sure about that. @Robert Raharjo do you know if scaleDown is supported?

Robert Raharjo

10/20/2025, 8:49 PM

Yes, it's supported. That doc refers to the tutorial. I'm not sure what you mean by the example is disabled. https://github.com/StarRocks/starrocks-kubernetes-operator/blob/main/doc/automatic_scaling_for_cn_nodes_howto.md This is another doc about the scaler (similar)

JD Baudean

10/20/2025, 9:31 PM

Thanks. All of those examples have the scaleDown policy as disabled.

Copy code

behavior: #  The scaling behavior is customized according to business scenarios, helping you achieve rapid or slow scaling or disable scaling.
          scaleUp:
            policies:
              - type: Pods
                value: 1
                periodSeconds: 10
          scaleDown:
            selectPolicy: Disabled

My CNs will not scale in until I add something like this

Copy code

scaleDown:  
            selectPolicy: Min
            stabilizationWindowSeconds: 300 
            policies:
            - type: Pods
              value: 1
              periodSeconds: 300

That works, but it doesn't always remove the CN from starrocks itself. Sometimes it cleans the old CN up fine, other times I'll still see the CN in the UI under

compute_nodes

(or when running a similar DB query) but it will show the node with an Error message like below, the only way to clear that out is to run the sql ALTER DB DROP COMPUTE.... command

Copy code

java.net.UnknownHostException: kube-starrocks-cn-10.kube-starrocks-cn-search.kube-starrocks.svc.cluster.local

JD Baudean

10/20/2025, 10:07 PM

actually I think I'm narrowing down the strange behavior, and it's not unique to the HPA. This only happens when I have 10 or more nodes in my statefulset and then remove nodes, either via the HPA or decreasing the replicas using

kubectl scale statefulset

For example if I have 15 CNs, starting with kube-starrocks-cn-0 to kube-starrocks-cn-14 and I remove one by changing the replicas from 15 to 14, the kube-starrocks-cn-14 pod is terminated, but looking at the compute nodes in the UI kube-starrocks-cn-9 is removed, and kube-starrocks-cn-14 is still in the UI with the java connection error. The same thing happens as I continue to scale down, it'll terminate pod cn-13 but remove cn-8 from starrocks itself. If I start with 10 or less nodes, with cn-9 being the latest, everything works fine.

JD Baudean

10/20/2025, 10:22 PM

I just brought my cluster down to 5 nodes from 15. The pods I have running are cn-0 to cn-4. But if I look in the UI or DB starrocks thinks I have nodes cn-0, cn-1, cn-10, cn-11, and cn-12. The last 3 show as down with java connection errors which makes sense, since those pods don't actually exist anymore. The only way to get cn-2, cn-3, and cn-4 to show back up in starrocks is to run the ALTER DROP COMPUTE sql command on cn-10, cn-11, and cn-12, and then delete pods cn-2, cn-3, and cn-4 forcing them to restart.

JD Baudean

10/20/2025, 10:23 PM

We are using PVCs for the log and cache volumes for each CN pod if that makes a difference

Ray Yan

10/20/2025, 10:57 PM

“I’m still having to manually DROP the CN from the database, which pretty much defeats the purpose of autoscaling.” Which Operator and CRD version are you using?

Ray Yan

10/20/2025, 11:02 PM

Thank you for your response. If you are using the latest operator(help you drop CN node from SR), it seems there is a bug in the Operator.

JD Baudean

10/21/2025, 1:45 PM

I'm using operator v1.11.2 and Starrocks version 3.5. I did see that bug in v1.10.2 with all nodes, now it only appears to be nodes in the double digits. This is the original bug that I thought was fixed in 1.11.2 until I had more than 10 CNs - https://github.com/StarRocks/starrocks/issues/62915. Do you know if there'a an open bug report for this?

Ray Yan

10/21/2025, 4:17 PM

The Operator only recently began supporting the ALTER DROP COMPUTE feature, so there is no corresponding issue yet. We would greatly appreciate it if you could report this issue. I will try to reproduce the error from my end, and fix it ASAP.

JD Baudean

10/21/2025, 4:52 PM

@Ray Yan Great thank you! Here's the issue, please let me know if you need any more info or if there's anything I can do to help. https://github.com/StarRocks/starrocks-kubernetes-operator/issues/713

Ray Yan

10/21/2025, 5:41 PM

I have submitted a PR(https://github.com/StarRocks/starrocks-kubernetes-operator/pull/714). If you have time, welcome to review it.

Ray Yan

10/22/2025, 6:18 PM

@JD Baudean Thank you again for your issue, and we have published a new version, v1.11.3

Open in Slack

Previous Next