@Rocky While trying to setup metrics using the the blog post part 1, it mentions

node_info{type="be_node_num", job="$job_name",state="dead"}

being "Number of failed BE nodes". However, this metric does not seem to be reported by the BE (we checked manually on the

/metrics

endpoint). Where is the metric coming from?

The metric

node_info{type="be_node_num", ...}

is not reported by the BE. It is a cluster-level metric that is reported by the Frontend (FE) or synthesized by the StarRocks Exporter. Here is why you don't see it on the BE

/metrics

endpoint and where to find it: 1. It is an FE-level metric Since the status of all Backend (BE) nodes (whether they are "alive" or "dead") is managed by the FE's heartbeat mechanism, only the FE can report the total count of failed BEs. * Check your *FE*'s metrics endpoint instead:

http://<fe_ip>:<fe_http_port>/metrics

(usually port

8030

). * In newer versions of StarRocks, cluster-wide status is often exported by the FE with names like

starrocks_fe_backend_dead_num

or

starrocks_fe_backend_num

. 2. It is provided by the

starrocks-monitor

Exporter The metric format

node_info{type="be_node_num", state="dead"}

is characteristic of the StarRocks Exporter (found in the starrocks-monitor repository). This exporter is a separate component (often deployed via Helm or as a standalone binary) that: 1. Connects to the StarRocks cluster (via SQL or the FE HTTP API). 2. Runs commands like

SHOW BACKENDS

. 3. Transforms the output into the Prometheus-compatible

node_info

metric. How to get this data If you are scraping FE/BE endpoints directly and don't see

node_info

, you can achieve the same alert using native FE metrics (depending on your version): * Total dead BEs:

starrocks_fe_backend_dead_num

* Total alive BEs:

starrocks_fe_backend_alive_num

If you are following the blog post's specific alerting rules, ensure you have deployed the StarRocks Exporter included in the

starrocks-monitor

suite, as those specific labels (

type="be_node_num"

) are often unique to that exporter's mapping. References • administration/management/monitoring/metrics.md • administration/management/BE_configuration.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/information_schema/be_metrics.md

@Rocky https://github.com/StarRocks/starrocks-monitor is a 404 Not Found FYI

The metric

node_info{type="be_node_num", ...}

is not reported by the BE binary directly. It is a cluster-management metric that is either synthesized by a monitoring sidecar or reported by the Frontend (FE). Here is where it comes from and how to find it: 1. It is provided by the StarRocks Exporter / Operator The specific label structure

node_info{type="be_node_num", state="dead"}

is characteristic of the StarRocks Exporter. * Kubernetes: If you are using the StarRocks Kubernetes Operator, the operator itself (or its included metrics exporter) scrapes the cluster state via SQL (e.g.,

SHOW BACKENDS

) and transforms it into this metric. * Bare Metal: Older guides referred to a standalone Prometheus exporter (previously in the

starrocks-monitor

repo) that performed this transformation. 2. The

starrocks-monitor

repository You are correct that the old

starrocks-monitor

repository is no longer active (404). Most official monitoring assets have been moved: * Dashboards: Now located in the main repository under extra/grafana. * Kubernetes Monitoring: Managed within the starrocks-kubernetes-operator repository. 3. Native FE Metrics (Workaround) If you are scraping FE and BE endpoints directly and do not see

node_info

, you should use the native FE metrics which report the same data. The FE is the only component that knows the "Dead/Alive" status of all BEs via its heartbeat mechanism. Check your FE metrics endpoint (

http://<fe_ip>:8030/metrics

) for: * `starrocks_fe_backend_dead_num`: The number of dead BE nodes. * `starrocks_fe_backend_alive_num`: The number of alive BE nodes. Summary The blog post assumes you are using the StarRocks monitoring stack (which includes an exporter/operator to create those specific labels). If you are building your own Prometheus config from scratch: 1. Scrape the FE to get the count of dead/alive backends. 2. Scrape the BEs for process-level stats (CPU, Memory, Disk). References • administration/management/monitoring/metrics.md • administration/management/monitoring/manage_alerts.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • sql-reference/information_schema/be_metrics.md