Hi all, we're running into some issues with scalin...
# questions-and-troubleshooting
j
Hi all, we're running into some issues with scaling down while running Starrocks 3.5 shared data in Azure AKS. Scaling up and down using the HPA generally works, but we've found that when the cluster scales down it will break running jobs/queries because it'll scale in a CN that's currently being used by the query... if that makes sense. I think the problem is that the HPA only looks at CPU/Memory resources, and doesn't care or even know if the CN is being used for running queries. This is what my HPA config looks like
Copy code
autoScalingPolicy: #DEPLOY-VALUE
      # you can select different versions of HPA (Horizontal Pod Autoscaler) based on the Kubernetes version you are
      # using to ensure compatibility and adaptability. the default version is v2beta2. #DEPLOY-VALUE: v2beta2 is OBE!
      maxReplicas: 10 # 10 #DEPLOY-VALUE - Can increase post-deploy by updating this value and running 'helm upgrade'
      minReplicas: 5
      hpaPolicy: #DEPLOY-VALUE
        metrics:
        - type: Resource
          resource:
            name: memory
            target:
              averageUtilization: 60
              type: Utilization
        - type: Resource
          resource:
            name: cpu
            target:
              averageUtilization: 60
              type: Utilization
        behavior:
          scaleUp:
            policies:
            - type: Pods
              value: 2
              periodSeconds: 20
          scaleDown:
            #selectPolicy: Disabled 
            # The stabilization window is the time HPA must wait after a scale-up
            # or before a scale-down, ensuring the low utilization is sustained.
            stabilizationWindowSeconds: 300  # 5 minutes minimum recommended

            # Policies define HOW scale-down happens. You can scale down by
            # a percentage of current pods, or a fixed number of pods.
            policies:
              - type: Pods
                value: 1  # Reduce 1 pod at a time
                periodSeconds: 300 # Reduce pods by 1 every 5 minutes
            selectPolicy: Min    # Apply the Min policy (type = Pods)
These are the kinds of errors we're seeing with etl queries when the cluster scales in
Copy code
"Failed to query StarRocks: 5609 (22000): Memory of query_pool exceed limit. try consume:81920 Backend: kube-starrocks-cn-0.kube-starrocks-cn-search.kube-starrocks.svc.cluster.local, Used: 150237386664, Limit: 150289495621. Mem usage has exceed the limit of query pool: BE:17895478",
Copy code
"Failed to query StarRocks: 5609 (22000): [E111]Fail to connect Socket{id=1428 addr=10.110.3.151:8060} (0x0x7f4702753ac0): Connection refused [R1][E112]Not connected to 10.110.3.151:8060 yet, server_id=1428 [R2][E112]Not connected to 10.110.3.151:8060 yet, server_id=1428 [R3][E112]Not connected to 10.110.3.151:8060 yet, server_id=1428",
Copy code
"Failed to query StarRocks: 1064 (HY000): starlet err grpc.GetShard(shardId=18699872) error: Deadline Exceeded: BE:2367208",