@Rocky Can the K8S operator expose useful Prometheus metrics ? How do you enable that in the helm chart values.yaml file ?

Yes, the StarRocks Kubernetes (K8S) Operator exposes Prometheus metrics related to its own performance (such as reconciliation counts, errors, and latencies), and it also provides a way to automatically enable metric scraping for the StarRocks clusters it manages. 1. Enabling Metrics for the Operator Itself The StarRocks Operator is built using standard operator frameworks that expose performance metrics on a dedicated port (typically port 8080). If you have the Prometheus Operator installed, you can enable a

ServiceMonitor

for the operator. In your

values.yaml

(for the

kube-starrocks

chart), you can enable it under the

operator

section:

yaml
operator:
  starrocksOperator:
    metrics:
      serviceMonitor:
        enabled: true
        # Optional: customize labels or interval
        # labels: 
        #   release: prometheus-stack
        # interval: 15s

2. Enabling Metrics for the StarRocks Cluster (FE/BE/CN) Most users also want to monitor the StarRocks database components (FE, BE, CN). The operator can create

ServiceMonitor

resources for these components or add the necessary Prometheus annotations. Using ServiceMonitor (Recommended for Prometheus Operator) Starting from chart version v1.8.4, you can enable

ServiceMonitor

for the entire cluster in the

starrocks

subchart section:

yaml
starrocks:
  metrics:
    serviceMonitor:
      enabled: true

Using Annotations (Standard Prometheus) If you are not using the Prometheus Operator, you can add annotations to the FE and BE services to allow Prometheus to auto-discover them:

yaml
starrocks:
  starrocksFESpec:
    service:
      annotations:
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8030"
  starrocksBeSpec:
    service:
      annotations:
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8040"

Key Metrics Exposed Once enabled, you will have access to: * Operator Metrics:

controller_runtime_reconcile_total

,

workqueue_depth

, and

controller_runtime_reconcile_errors_total

. * Cluster Metrics:

starrocks_fe_query_latency_ms

,

starrocks_be_compaction_score

,

starrocks_be_disks_usage_bytes

, and more. Note: Ensure your Prometheus instance has the necessary permissions (RBAC) to scrape across the namespaces if your operator and cluster are in different namespaces. References • administration/management/monitoring/Monitor_and_Alert.md • quick_start/helm.md • administration/management/monitoring/metrics-materialized_view.md

Point 1 does not work, the chart does not seem to have anything for operator service monitors.

Yes, the recommendations provided are in line with the integration-prometheus-grafana.md documentation in the official repository. The document focuses on two main methods for enabling metrics collection for the StarRocks Cluster (FE and BE). Here is how they map to the

values.yaml

file: 1. Using ServiceMonitor (Recommended) As specified in section 2.2 of the GitHub document, if you are using the Prometheus Operator, you can enable a

ServiceMonitor

resource. In your

values.yaml

, this is done within the

starrocks

section:

yaml
starrocks:
  metrics:
    serviceMonitor:
      enabled: true

Alignment: This is the standard way the Helm chart (v1.7.1+) automates the creation of the

ServiceMonitor

CRD, which the Prometheus Operator then uses to discover the FE and BE targets. 2. Using Annotations As specified in section 2.1 of the GitHub document, if you are using a standard Prometheus deployment (without the operator), you can add annotations directly to the FE and BE services:

yaml
starrocks:
  starrocksFESpec:
    service:
      annotations:
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8030"
  starrocksBeSpec:
    service:
      annotations:
        <http://prometheus.io/scrape|prometheus.io/scrape>: "true"
        <http://prometheus.io/path|prometheus.io/path>: "/metrics"
        <http://prometheus.io/port|prometheus.io/port>: "8040"

Alignment: These annotations match the standard Prometheus scraping configuration described in the "Turn on by adding annotations" section of the guide. 3. Operator Metrics (Self-Monitoring) While the

integration-prometheus-grafana.md

file focuses primarily on the database cluster (FE/BE), the StarRocks Operator itself also exposes metrics (reconciliation status, workqueue depth, etc.). In the

kube-starrocks

umbrella chart, you can enable these separately. This is often found under the operator sub-section:

yaml
operator:
  starrocksOperator:
    metrics:
      serviceMonitor:
        enabled: true

Summary of Alignment * Cluster Metrics: My previous advice matches sections 2.1 and 2.2 of the GitHub doc exactly. * Operator Metrics: This is a feature of the operator's own codebase (standard for K8s operators) and is supported by the Helm chart, though the document you linked is specifically tailored to getting the FE/BE database metrics into Grafana. Recommendation: If you are using the StarRocks Grafana Dashboard, the ServiceMonitor approach is the most robust as it handles the dynamic IPs of FE and BE pods automatically. References • administration/management/monitoring/Monitor_and_Alert.md • quick_start/helm.md

@Moreno Garcia So what I found is that adding prometheus label on the operator pod for the /metrics 8080 endpoint is not enough in our VictoriaMetrics based envirnment. Apparently the vm agent requires the

Ports:

to be exposed from what I read in our internal docs by our infra/monitoring team. This means it's only missing something like

containers:
        - name: manager
          ports:
            - name: metrics
              containerPort: 8080
              protocol: TCP

Is this something you'd consider adding to the public operator helm chart ? Adding

metrics:
      serviceMonitor:
        enabled: true

to the starrocksOperator does not seem to work, it's not something the operator chart seems to handlde