@Rocky hi rocky StarRocks Kubernetes Operator v1.11.4 on K8s 1.30, shared_data mode, CN with native HPA configured via starRocksCnSpec.autoScalingPolicy (memory utilization target). HPA correctly scales the StarRocksCluster CR — kubectl shows SRC.spec.starRocksCnSpec.replicas updated from 1 to 2, HPA status desired=2, but the underlying StatefulSet.spec.replicas stays at 1. Operator log shows the last reconcile for this SRC was before the HPA scale event, and no further reconcile fires for 2+ minutes despite the spec change via the /scale subresource. SRC.metadata.generation increments to 2 but status.observedGeneration remains empty, suggesting the watcher is not receiving or filtering out spec-only updates from the autoscaler. Touching any annotation on the SRC immediately triggers a reconcile and the StatefulSet then scales correctly within seconds — is this a known watch/predicate issue with HPA-driven /scale updates in v1.11.x, and what's the recommended fix or operator version that handles this end-to-end?

Based on your description, you are likely encountering a known issue in the StarRocks Kubernetes Operator v1.11.x related to how the controller filters events from the Kubernetes API server. Analysis of the Issue The symptoms you described—`metadata.generation` incrementing while

status.observedGeneration

remains empty and no reconcile firing—point to a Predicate filtering issue in the operator's controller-runtime logic. 1. Predicate Filtering: The operator uses a

GenerationChangedPredicate

(or similar logic) to ignore updates that only affect the

status

block. When the HPA updates the

/scale

subresource, it modifies

spec.starRocksCnSpec.replicas

. While this increments the

metadata.generation

, some versions of the operator had logic that failed to recognize this specific nested field change as a "reconcileable" event unless other metadata (like annotations) also changed. 2. Observed Generation: The fact that

status.observedGeneration

remains empty suggests that the operator has never successfully completed a status update for that CR, or the status subresource is not being updated correctly during the reconcile loop. 3. HPA & Shared Data: In

shared_data

mode, the operator must handle CN scaling specifically. There was a known issue (tracked similarly to Issue #725 and related PRs) where the operator's internal "diff" logic between the desired and actual state of the StatefulSet did not always trigger correctly when the change originated from the HPA subresource. Recommended Fix & Version This behavior was identified and addressed in updates following the v1.11.4 release. * Upgrade to v1.12.x: It is highly recommended to upgrade to v1.12.0 or later. The v1.12.x branch includes significant refactoring of the reconciliation and status update logic, specifically improving how the operator handles subresource updates and HPA-driven scaling for CN nodes. * CRD Update Requirement: Note that starting from v1.11.2, a new field

podManagementPolicy

was added to the CRD. If you haven't manually updated your CRD to the latest version, the operator might fail to reconcile correctly because it cannot process the full schema, which often leads to the "empty status" symptom you are seeing. Ensure you apply the latest CRDs:

bash
    kubectl apply -f <https://github.com/StarRocks/starrocks-kubernetes-operator/releases/download/v1.12.0/starrocks.com_starrocksclusters.yaml>

Immediate Workarounds If you cannot upgrade immediately, you can use the following workarounds: * Touch Annotations: As you discovered, adding/modifying an annotation (e.g.,

kubectl annotate src <name> reconcile-trigger=$(date +%s) --overwrite

) forces the predicate to pass and triggers the scale. * Verify HPA Target: Ensure your HPA is targeting the

StarRocksCluster

(SRC) and not the underlying

StatefulSet

directly. The operator is designed to manage the StatefulSet replicas based on the SRC spec; if HPA targets the STS, the operator will likely fight it and scale it back. * Check Operator Permissions: Ensure the operator's

ClusterRole

has the necessary permissions for the

scale

subresource on `starrocksclusters`:

yaml
    • apiGroups: ["<http://starrocks.com|starrocks.com>"]
      resources: ["starrocksclusters/scale"]
      verbs: ["get", "update", "patch"]

Summary: The issue is a known watch/predicate limitation in v1.11.4. Upgrading to v1.12.x is the definitive fix, as it matures the HPA support and ensures that

/scale

subresource updates correctly trigger the reconciliation loop. References • deployment/sr_operator.md • quick_start/helm.md • deployment/helm.md