http://starrocks.io logo
Join Slack
Powered by
Hello, I was wondering if our team is doing someth...
# questions-and-troubleshooting
w
Hello, I was wondering if our team is doing something wrong regarding scaling StarRocks for highly concurrent scenarios. We are scaling the cluster in shared-data mode to twice the number of compute and FE nodes (3->7 for each) and using S3 for the object store, but not seeing any increase in query throughput after scaling. We are self-hosting StarRocks 3.5. So far we've followed most of the recommended recommendations in https://www.starrocks.io/blog/starrocks-best-practices-queries • We've increased the number of max_user_connections from the default of 100 to 1000 • Enabled the datacache • Minimized Selects to a couple columns Now what's interesting is that there seems to be little usage on resources on FE and BE nodes. When we scale the nodes up, the resource usage does drop, but the actual QPS the cluster can process/is processing seems to be about the same. Is there some setting we're missing or some throttling happening that we're unaware of?
j
This doesn't address your question, but I'm curious how you're handling scaling back down? As far as I know only scaling up with the HPA is supported, while scaling back down is not? And you also have to run an 
ALTER
DB command to remove the scaled down CNs
w
@JD Baudean We've only tried this with scaling up, but that's good to know and we'll keep in mind. Could you point us to where you saw documentation regarding issues when scaling down with HPA?
j
Yea let me see if I can find it. Here's the doc for manually scaling in a CN node and the DB command you have to run. https://docs.starrocks.io/docs/3.5/administration/management/Scale_up_down/#scale-cn-in
Actually I might have been mistaken. I couldn't find specifics in the Starrocks documentation (if you find it please let me know) but I asked AI and it gave me a scale down example to put in my helm chart and it appeared to work. I need to tune it more but you get the idea. I'm pasting the entire HPA section but the only part I changed was the scaleDown section
Copy code
autoScalingPolicy: #DEPLOY-VALUE
      # you can select different versions of HPA (Horizontal Pod Autoscaler) based on the Kubernetes version you are
      # using to ensure compatibility and adaptability. the default version is v2beta2. #DEPLOY-VALUE: v2beta2 is OBE!
      maxReplicas: 10 #DEPLOY-VALUE - Can increase post-deploy by updating this value and running 'helm upgrade'
      minReplicas: 1
      hpaPolicy: #DEPLOY-VALUE
        metrics:
        - type: Resource
          resource:
            name: memory
            target:
              averageUtilization: 30
              type: Utilization
        - type: Resource
          resource:
            name: cpu
            target:
              averageUtilization: 30
              type: Utilization
        behavior:
          scaleUp:
            policies:
            - type: Pods
              value: 1
              periodSeconds: 10
          scaleDown:
            # The stabilization window is the time HPA must wait after a scale-up
            # or before a scale-down, ensuring the low utilization is sustained.
            stabilizationWindowSeconds: 300  # 5 minutes minimum recommended

            # Policies define HOW scale-down happens. You can scale down by
            # a percentage of current pods, or a fixed number of pods.
            policies:
            - type: Percent
              value: 100            # Allow scaling down by up to 100% of replicas
              periodSeconds: 30     # Check every 30 seconds
            selectPolicy: Max       # Select the policy that allows the largest reduction
Ah, here is where I saw scaleDown disabled. And it is causing problems for us after turning it on overnight, I'm having to manually remove the nodes with the DB command when it scales in - https://docs.starrocks.io/docs/deployment/sr_operator/#automatic-scaling-for-cn-cluster
Open in Slack
PreviousNext
Powered by