http://starrocks.io logo
Join Slack
Powered by
hey <@U0A71G31CDV> I have a starrocks shared data ...
# questions-and-troubleshooting
v
hey @Rocky I have a starrocks shared data cluster where I am not able to do the vertical scaling for CN pods. anything you can suggest here?
r
Vertical scaling of CN (Compute Node) pods in a StarRocks shared-data cluster on Kubernetes is typically handled by updating the StarRocksCluster (SRC) custom resource. If you are unable to perform vertical scaling (changing CPU/Memory resources), here are the most common reasons and how to resolve them: 1. Update the Custom Resource (not the Deployment) The StarRocks Operator manages the underlying Kubernetes Deployments/StatefulSets. If you try to 
kubectl edit deployment
directly, the Operator will likely overwrite your changes back to the original state defined in the 
StarRocksCluster
resource. Correct way: Edit your StarRocksCluster YAML:
Copy code
bash
kubectl edit starrockscluster <cluster_name> -n <namespace>
Update the 
resources
section under `starRocksCnSpec`:
Copy code
yaml
spec:
  starRocksCnSpec:
    requests:
      cpu: "8"
      memory: "32Gi"
    limits:
      cpu: "16"
      memory: "64Gi"
2. Check for 
autoScalingPolicy
Conflicts If you have an 
autoScalingPolicy
(HPA) configured in your Spec, it might interfere with manual resource updates depending on the Operator version. * Check if 
autoScalingPolicy
is defined under 
starRocksCnSpec
. _ If you are trying to change the _number* of replicas while HPA is on, you must remove the 
replicas
field from the spec as the Operator/HPA takes over. 3. StarRocks Internal Memory Limits (
mem_limit
) CN pods automatically attempt to calculate their memory usage based on the pod's cgroup limits. However, if you have a hardcoded 
mem_limit
in your 
cn.conf
(via 
configMap
or 
env
variables), StarRocks will ignore the additional memory provided by the vertical scale. * Recommendation: Ensure 
mem_limit
is not manually set in your configuration so StarRocks can default to ~90% of the pod's memory limit. 4. Kubernetes Node Capacity and Quotas If you update the 
requests
and the pods stay in 
Pending
state: * Check for Resource Quotas: Run 
kubectl describe quota -n <namespace>
to see if your namespace has reached its limit. * Check Node Capacity: Ensure your physical/virtual nodes have enough unallocated CPU/Memory to host the larger CN pods. 5. Operator Logs If the pods are not restarting after you update the 
StarRocksCluster
CR, the Operator might be encountering an error. Check the logs:
Copy code
bash
kubectl logs -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator> -n <operator_namespace>
Quick Troubleshooting Questions: * Are you using the StarRocks Operator? If so, what version? * Do the pods attempt to restart (Rolling Update) after you apply the change, or do they stay on the old version? * What is the status of the 
StarRocksCluster
resource? (
kubectl get src
) References • deployment/sr_operator.md • deployment/upgrade.md • administration/management/Scale_up_down.md
v
@Rocky below is my CN specs
Copy code
starRocksCnSpec:
    image: starrocks/cn-ubuntu:4.0.5
    replicas: 4
    limits:
      cpu: 1700m
      memory: 6Gi
    requests:
      cpu: 1700m
      memory: 6Gi
    storageVolumes:
    - name: cn-cache-data
      storageClassName: "starrocks"
      hostPath:
        path: /storage
      storageSize: 20Gi
      mountPath: /opt/starrocks/cn/storage
    configMapInfo:
      configMapName: starrocks-olap-cn-cm
      resolveKey: cn.conf
    serviceAccount: "starrocks"
    schedulerName: default-scheduler
    podLabels:
      clusterName: starrocks-olap
      group: cn
    terminationGracePeriodSeconds: 180
    cnEnvVars:
    - name: "MYSQL_PWD"
      valueFrom:
        secretKeyRef:
          name: starrocks-olap-mysql-root-pw
          key: STARROCKS_MYSQL_PWD
    - name: LOG_CONSOLE
      value: "1"
    tolerations:
      - key: eksServiceName
        operator: Equal
        value: starrocks-olap
        effect: NoSchedule
    affinity:
      nodeAffinity:
        requiredDuringSchedulingIgnoredDuringExecution:
          nodeSelectorTerms:
            - matchExpressions:
                - key: eksServiceName
                  operator: In
                  values:
                    - starrocks-olap
I updated the requests and limits here and did a 
kubectl apply -f cluster.yaml
but the starrockscluster resource goes in 
reconciling
state
r
When the 
StarRocksCluster
resource stays in the 
reconciling
state after an update, it typically means the StarRocks Operator is attempting to roll out the changes but is encountering an issue that prevents the new Pods from reaching a "Ready" state. Since you updated the CPU/Memory requests and limits, the most common reason for a stuck reconciliation is that the Kubernetes scheduler cannot find nodes with enough available resources to satisfy the new requirements. Here is a step-by-step guide to troubleshoot and resolve this: 1. Check Pod Status and Events First, identify if the CN Pods are actually being created and why they might be stuck (e.g., 
Pending
).
Copy code
bash
# List the CN pods
kubectl get pods -l clusterName=starrocks-olap,group=cn

# Check events for a specific CN pod (especially if it's in Pending or CrashLoopBackOff)
kubectl describe pod <cn-pod-name>
Common issues to look for in events: * 
FailedScheduling
: "0/N nodes are available: ... insufficient cpu/memory." This means your nodes don't have enough allocatable space for 
1700m
CPU or 
6Gi
RAM. * 
CrashLoopBackOff
: The process inside the container is failing. Check the logs: 
kubectl logs <cn-pod-name>
. 2. Check Operator Logs The Operator is responsible for the transition. If it hits an error, it will log it.
Copy code
bash
# Find the operator pod name
kubectl get pods -n <operator-namespace> -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator>

# Check the logs for reconciliation errors
kubectl logs -f -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator> -n <operator-namespace>
3. Check the StarRocksCluster Resource The status of the CRD itself might provide more context:
Copy code
bash
kubectl describe src <your-cluster-name>
Look at the 
Status
and 
Events
sections at the bottom. 4. Common Pitfalls with your Spec * Node Capacity: You have 
replicas: 4
with 
requests
of 
1700m
CPU and 
6Gi
RAM. Total requirement: ~7 CPUs and 24Gi RAM. Ensure your nodes labeled 
eksServiceName: starrocks-olap
have this much available capacity after accounting for system overhead and other pods. * HostPath Storage: You are using 
hostPath
. If a Pod is deleted and recreated on a different node (or if the node is full), and the 
hostPath
directory 
/storage
doesn't exist or has permission issues on the new node, the Pod will fail to start. _ _Note:* Ensure the 
serviceAccount: "starrocks"
has the necessary permissions to write to that host path. * Termination Grace Period: You set 
terminationGracePeriodSeconds: 180
. During a rolling update, the operator waits for the old pod to shut down. If the CN process takes a long time to exit, the reconciliation will stay in progress for at least 3 minutes per pod. Recommended Action If the pods are 
Pending
due to 
insufficient cpu
, you may need to: 1. Increase the size/number of your EKS nodes. 2. Or, reduce the 
requests
to a level that fits your current node size. How to "unstick" it: If you realize the specs are too high for your cluster, you can edit the YAML again with lower values and 
kubectl apply
it. The operator will pick up the new desired state and try to reconcile again. References • quick_start/helm.md • deployment/sr_operator.md • deployment/upgrade.md
v
@Rocky the behavior of the CN pod is like after resource update: • the last CN pod gets terminated and never comes up • the cluster goes in 
reconciling
state and never comes back to running state • I do not see pods pending, failed or any other state. seems something breaks silently
r
It sounds like the StarRocks Operator is encountering an issue while managing the lifecycle of the CN pods during a rolling update. When a cluster stays in the 
reconciling
state and pods "break silently," it usually indicates that the Operator's control loop is stuck or encountering an error it cannot recover from. Here are the recommended steps to troubleshoot and resolve this issue: 1. Check StarRocks Operator Logs This is the most critical step. The Operator logs will reveal why it isn't creating the next pod or why it's stuck in the reconciliation loop.
Copy code
bash
kubectl logs -n <namespace> -l <http://app.kubernetes.io/name=starrocks-operator|app.kubernetes.io/name=starrocks-operator>
Look for errors related to: * StatefulSet/Deployment updates: Errors like 
forbidden: updates to statefulset spec for fields other than...
* FE Communication: The Operator often needs to communicate with the FE to register or deregister nodes. If it can't reach the FE, it may hang. * Resource Validation: Check if there are errors indicating the resource update (CPU/Memory) is invalid for the current environment. 2. Inspect the StarRocksCluster (SRC) Resource Check the status and events of your StarRocks cluster custom resource. It might provide a hint about which component is failing.
Copy code
bash
kubectl describe src <your-cluster-name> -n <namespace>
Look at the 
Status
section and the 
Events
at the bottom. 3. Check FE Node Status If the last CN pod was terminated, the FE might still think it is "Alive." Log into your StarRocks cluster (via MySQL client) and check the status of the compute nodes:
Copy code
sql
SHOW COMPUTE NODES;
If nodes are stuck in a 
Decommission
state or if the FE is unreachable, the Operator might wait indefinitely for the FE to signal that it's safe to proceed. 4. Verify the Underlying Controller Depending on your version, CN nodes are managed via a 
StatefulSet
or 
Deployment
. Check if that resource exists and if it has any errors:
Copy code
bash
kubectl get statefulset -l "<http://starrocks.com/node-type=cn|starrocks.com/node-type=cn>" -n <namespace>
# OR
kubectl get deployment -l "<http://starrocks.com/node-type=cn|starrocks.com/node-type=cn>" -n <namespace>
If the resource is present but the 
READY
count is 0/0 and the 
DESIRED
count is also 0, the Operator has scaled it down but is failing to scale it back up with the new configuration. 5. Common Causes for "Silent Breaks" * Finalizers: The Operator uses finalizers to ensure graceful shutdown. If a finalizer on the Pod or the SRC resource is stuck, the object won't be deleted or recreated. * Invalid Resource Requests: If you updated resources to values that are impossible to satisfy (e.g., requesting more memory than available on any node), and you have a 
PriorityClass
or specific 
ResourceQuota
, sometimes pods are not even created by the controller. * Operator Version Mismatch: Ensure your StarRocks Operator version is compatible with the StarRocks image version you are using. Next Step: If you can provide the output of 
kubectl logs
for the operator or the 
Status
section of 
kubectl get src <name> -o yaml
, I can help identify the specific error. References • deployment/deploy_manually.md • administration/management/logs.md • deployment/sr_operator.md • quick_start/helm.md • deployment/helm.md
v
@Rocky I see below in the output of 
kubectl describe src starrocks-olap
command. does that mean CN won't work on Vertical scaling?
Copy code
Status:
  Phase:  running
  Star Rocks Cn Status:
    Horizontal Scaler:
    Phase:     running
    Replicas:  4
    Resource Names:
      starrocks-olap-cn
    Running Instances:
      starrocks-olap-cn-3
      starrocks-olap-cn-1
      starrocks-olap-cn-0
      starrocks-olap-cn-2
    Selector:      <http://app.kubernetes.io/component=cn,app.starrocks.ownerreference/name=starrocks-olap-cn|app.kubernetes.io/component=cn,app.starrocks.ownerreference/name=starrocks-olap-cn>
    Service Name:  starrocks-olap-cn-service
r
The status message you are seeing in 
kubectl describe src
refers specifically to the Horizontal Pod Autoscaler (HPA) configuration for the Compute Nodes (CN). It does not mean that CNs do not support vertical scaling. In the context of the StarRocks Operator, scaling works as follows: 1. Horizontal Scaling The "Horizontal Scaler" section in the output indicates the status of the HPA (Horizontal Pod Autoscaler). If you have configured an 
autoScalingPolicy
in your StarRocksCluster (SRC) spec, the operator manages the number of replicas dynamically based on CPU or memory usage. 2. Vertical Scaling Vertical scaling (adjusting CPU and Memory resources) is fully supported for CNs. To scale vertically, you simply update the 
resources
(or 
requests
and 
limits
) in the 
starRocksCnSpec
section of your StarRocksCluster YAML:
Copy code
yaml
spec:
  starRocksCnSpec:
    requests:
      cpu: "8"
      memory: "32Gi"
    limits:
      cpu: "16"
      memory: "64Gi"
How the Operator handles it: When you apply a change to the resource requests or limits: * The StarRocks Operator detects the change in the specification. * It performs a rolling update of the CN pods. * It replaces the existing pods one by one with new pods that have the updated resource configurations, ensuring high availability during the process. Summary The presence of "Horizontal Scaler" in the status is just to provide visibility into the horizontal scaling component. You can still scale vertically by modifying the resource definitions in your CRD, and the Operator will manage the deployment of those changes automatically. References • deployment/sr_operator.md • deployment/upgrade.md • administration/management/Scale_up_down.md • deployment/deploy_manually.md
v
my bad, i made some human error. VPA is working
Open in Slack
PreviousNext
Powered by