@Rocky Does StarRocks expose a Prometheus metric for unhealthy tablet count, or is the only way to get this via SHOW PROC '/statistic'?

StarRocks does not currently expose a specific, built-in Prometheus metric for the "unhealthy tablet count." The standard and most reliable way to retrieve this information is indeed through the

SHOW PROC

commands: * Cluster-wide count:

SHOW PROC '/statistic';

returns the

UnhealthyTabletNum

and

InconsistentTabletNum

for the entire cluster. * Database-specific count:

SHOW PROC '/statistic/<db_id>';

provides the count and the specific IDs of unhealthy tablets within a particular database. Alternative Methods and Workarounds If you need to monitor this via Prometheus or external systems, you have a few options: 1. SQL Queries via `information_schema`: While there is no direct "unhealthy" flag in the metadata tables that perfectly matches the internal FE logic, you can query for tablets that might be problematic (e.g., those with too many versions):

sql
    SELECT BE_ID, TABLET_ID
    FROM information_schema.be_tablets
    WHERE NUM_ROWSET > 700; -- High version count often leads to unhealthy status

2. Custom Exporters: Many users in the StarRocks community use a custom script or a generic SQL exporter (like

sql_exporter

) to execute

SHOW PROC '/statistic'

or query the

information_schema

and expose the results as a Prometheus gauge. 3. Audit Logs / FE Logs: The FE logs will often contain "tablet is not normal" warnings, but this is not a quantitative metric. Why it isn't in Prometheus StarRocks FE metrics (exported at

:8030/metrics

) focus on instantaneous state and cumulative performance (latencies, request counts, etc.). The "unhealthy tablet" status is a derived state calculated by the Tablet Checker and Tablet Scheduler daemons in the FE, which is currently exposed through the

PROC

interface rather than the standard metric repository. For more details on diagnosing specific tablet issues, you can refer to the official Manage Replica documentation. References • administration/management/monitoring/metrics.md

@Rocky is there a specific metric or metrics that show the Cluster health is YELLOW? meaning reads can still happen but some writes and other cluster operations can't, and tablets are being distributed

StarRocks does not have a native Prometheus metric that explicitly outputs a "YELLOW" health status. In StarRocks, cluster health is a derived state rather than a single exported variable. To monitor the conditions you described—where replicas are missing or being redistributed—you should use a combination of the following Prometheus metrics and administrative commands. 1. Prometheus Metrics for "Yellow" States The "Yellow" state in other systems typically corresponds to under-replicated tablets or active redistribution. You can monitor this in StarRocks via these metrics: Metric Name | Description | Significance

starrocks_fe_scheduled_pending_tablet_num

| Number of tablet clone tasks in Pending state. | Indicates tablets waiting to be repaired or moved.

starrocks_fe_scheduled_running_tablet_num

| Number of tablet clone tasks currently Running. | Shows active redistribution/repair (Clone tasks).

starrocks_fe_tablet_num

| Total number of tablets in the cluster. | Used as a baseline for ratios. Interpretation: * Active Redistribution: If

scheduled_running_tablet_num

is high, the cluster is actively balancing or repairing. * Write Failures: StarRocks requires a write quorum (usually a majority of replicas) to succeed. If the number of healthy replicas for a tablet drops below the quorum, writes to that tablet will fail, even if reads (from the remaining healthy replica) still work. 2. Why

SHOW PROC

is still required The internal "Tablet Checker" in the FE is the only component that maintains the definitive count of "unhealthy" vs "inconsistent" tablets. Prometheus metrics currently focus on the tasks created to fix those tablets (Clones) rather than the state of the tablets themselves. To see the exact number of tablets that are currently in a "Yellow" or "Red" state (missing replicas or versions), you must use:

sql
SHOW PROC '/statistic';

* UnhealthyTabletNum: Tablets that have fewer than the required number of replicas or have failed replicas. * InconsistentTabletNum: Tablets where replicas have diverging data versions. 3. Summary of Cluster States While StarRocks doesn't use the Red/Yellow/Green labels in its API, the behavior maps as follows: * Green: All replicas are healthy and balanced. * Yellow: Some replicas are missing or lagging, but a quorum still exists. Reads work; writes work. StarRocks is likely running

CLONE

tasks (check

starrocks_fe_scheduled_running_tablet_num

). * Red: Replicas have dropped below the write quorum or are entirely missing. Writes will fail with "tablet writer failed" or "timeout" errors. Recommendation: Most production users set up a custom exporter (like a script or

sql_exporter

) to query

SHOW PROC '/statistic'

and transform the

UnhealthyTabletNum

into a Prometheus gauge for alerting. References • administration/management/monitoring/metrics.md • administration/management/resource_management/Replica.md References • faq/operation_maintenance_faq.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md

@Rocky but am I right in assuming that those schedling and pending operations happen periodically even if the cluster is healthy?

You are correct that the Tablet Checker and Tablet Scheduler are background processes that run periodically. However, the metrics for pending and running operations (

starrocks_fe_scheduled_pending_tablet_num

and

starrocks_fe_scheduled_running_tablet_num

) should ideally be 0 in a perfectly healthy and balanced cluster. Here is the breakdown of why you might see non-zero values and what they mean for cluster health: 1. "Checks" vs. "Tasks" * The Checks (Background): The FE runs a Tablet Checker thread (defaulting to every few seconds/minutes) to scan the metadata. This is a low-overhead internal scan and does not increase the "pending/running" task metrics. * The Tasks (Metrics): A "Task" (Clone) is only created when the Checker finds something wrong. These tasks are what show up in your Prometheus metrics and

SHOW PROC

. 2. REPAIR vs. BALANCE When you see non-zero scheduling metrics, the tasks fall into two categories (viewable via

SHOW PROC '/cluster_balance/pending_tablets'

): * REPAIR Tasks (The "Yellow" State): These occur when a tablet has lost a replica (e.g., a BE went down) or a replica is corrupted/stale. In this state, the cluster is often "Yellow"—it is still functional because a quorum exists, but it is actively working to restore redundancy. * BALANCE Tasks (Healthy but Optimizing): These occur even if all tablets have the correct number of replicas. If one BE has significantly more data or more replicas than others, the Scheduler will move tablets to low-load BEs. This often happens after adding new BEs or after massive data ingestion/deletion. 3. Should they be zero? * In a steady, healthy state: These metrics should be 0. If your cluster is healthy, balanced, and no BEs have been added or removed recently, you should see no pending or running tasks. * In a "Healthy but Busy" state: You might see occasional BALANCE tasks if your data distribution is slightly uneven, but these should eventually finish and return to 0. * If they never reach 0: This usually indicates a problem, such as: * A BE is constantly reporting disk-full (preventing balance). * A specific tablet is "stuck" (cannot find a healthy source replica to clone from). * The

max_scheduling_tablets

limit is too low, causing a long backlog of balancing. Summary for Monitoring If you want to use these as "Yellow" health indicators: *

REPAIR

tasks = Yellow/Red Alert. Redundancy is compromised. *

BALANCE

tasks = Maintenance Alert. The cluster is healthy but busy re-organizing data. To see which type is running, you can use:

sql
SELECT TYPE, COUNT(*) 
FROM information_schema.fe_tablet_schedules 
GROUP BY TYPE;

References • administration/management/resource_management/Replica.md • sql-reference/information_schema/fe_tablet_schedules.md References • administration/management/resource_management/Replica.md • faq/operation_maintenance_faq.md • using_starrocks/Colocate_join.md • knowledge/admin_and_operation/shared_data_compaction.md

@Rocky but my point is, REPAIR alone is not a good indicator, others thing can create one of those tasks - not just a recovery from a node being down or something

You are absolutely correct. Monitoring only the presence of

REPAIR

tasks can be noisy because they are triggered by a variety of internal states, not just a "node down" event. To find a true "Yellow" state (degraded but functional), you need to look at the reason for the repair and the unhealthy tablet count. 1. Why

REPAIR

is triggered (Beyond Nodes Being Down) A

REPAIR

task is created whenever the FE's Tablet Checker detects that a tablet's actual state doesn't match its target state. This includes: *

VERSION_MISSING

/

REPLICA_VERSION_STALE

: A replica missed a load or is lagging behind. (Reads still work from other replicas; writes still work if quorum is met). *

REPLICA_MISSING

: A replica was lost due to disk failure or a BE being decommissioned. *

COLOCATE_REPLICA_MISSING

: For Colocate Tables, if a replica isn't on the specific BE mandated by the colocation group. *

REDUNDANT_REPLICA

: Too many replicas exist (often after a repair or a rebalance). The "repair" here is actually a deletion. 2. How to distinguish "Maintenance" from "Yellow Health" To get a better signal, you should query the

information_schema.fe_tablet_schedules

table. It contains a

SCHEDULE_REASON

column that tells you exactly why the task exists. SQL for Health Monitoring:

sql
SELECT 
    TYPE, 
    STATE, 
    SCHEDULE_REASON, 
    COUNT(*) as task_count
FROM information_schema.fe_tablet_schedules
GROUP BY TYPE, STATE, SCHEDULE_REASON;

* Yellow Signal: High count of

REPAIR

with reasons like

REPLICA_MISSING

or

VERSION_MISSING

. * Pure Maintenance Signal: High count of

BALANCE

tasks. This just means the cluster is moving data to even out disk usage or IO. 3. The most accurate "Yellow" indicator If you want one single number that defines "Yellow," it is the

UnhealthyTabletNum

from

SHOW PROC '/statistic'

. * Healthy (Green):

UnhealthyTabletNum = 0

. * Degraded (Yellow):

UnhealthyTabletNum > 0

AND

InconsistentTabletNum = 0

. (Redundancy is low, but data is likely still safe and readable). * Critical (Red):

UnhealthyTabletNum

is high AND you see errors in

information_schema.load_tracking_logs

or

fe.audit.log

showing write failures (meaning replicas dropped below quorum). Summary Recommendation Since StarRocks doesn't have a single

starrocks_cluster_health_status

metric, the best practice is to: 1. Monitor

UnhealthyTabletNum

via a custom script/exporter as your primary "Yellow" alert. 2. Use

fe_tablet_schedules

to differentiate between a critical repair (lost replicas) and a routine balance (moving data for efficiency). If you see

REPAIR

tasks but

UnhealthyTabletNum

is 0, the cluster is essentially healthy and just performing "cleanup" (like removing redundant replicas). References • administration/management/resource_management/Replica.md • sql-reference/information_schema/fe_tablet_schedules.md • administration/management/FE_configuration.md ) References • sql-reference/information_schema/fe_tablet_schedules.md • administration/management/FE_configuration.md • sql-reference/information_schema/be_tablets.md

@Rocky im was looking for prometheus metrics, not sql commands

StarRocks does not have a single native Prometheus metric that explicitly outputs a "YELLOW" status. In the StarRocks architecture, "Yellow" is a derived state—usually defined as tablets having missing replicas but still maintaining enough for a read quorum. Since you specifically want Prometheus metrics (not SQL), you should monitor the following combination to detect this state: 1. Active Repair and Balancing Metrics These are the most direct indicators that the cluster is currently "Yellow" (re-distributing or repairing data): Metric Name | Type | Significance for "Yellow" State

starrocks_fe_scheduled_pending_tablet_num

| Gauge | Tablets waiting to be repaired or moved.

starrocks_fe_scheduled_running_tablet_num

| Gauge | Tablets currently being cloned/repaired. * Interpretation: If these are non-zero, the cluster is in a state of flux. If

running

is high and stays high, it confirms your description of tablets being redistributed. 2. Write Health Metrics (The "Writes blocked" signal) When the cluster is in a severely degraded "Yellow" or "Red" state where writes fail but reads work (often because the write quorum is lost), monitor these: Metric Name | Type | Significance

starrocks_fe_tablet_num

| Gauge | Monitor for sudden drops in total tablets across specific BEs.

starrocks_be_engine_requests_total{type="load", status="failed"}

| Counter | Sudden spikes in load failures often indicate a loss of write quorum.

starrocks_fe_query_err_rate

| Gauge | Monitors the rate of query errors (useful if only some reads are failing). 3. Missing: The "Unhealthy Tablet Count" Crucially, the metric

UnhealthyTabletNum

(which is the gold standard for "Yellow") is not natively exported to Prometheus by the FE's default

/metrics

endpoint in most versions. It is only available via: *

SHOW PROC '/statistic';

* Querying

information_schema.fe_metrics

(if using a SQL-to-Prometheus exporter). Why this matters: The

starrocks_fe_scheduled_*

metrics only show tasks. If a tablet is unhealthy but the scheduler hasn't created a task for it yet (or the task failed), the "scheduled" metrics might be 0 while the cluster is still "Yellow." Recommendation for a "Yellow" Alert Most production setups create a "Yellow" alert in Prometheus/Grafana using this logic: 1. High Level:

starrocks_fe_scheduled_running_tablet_num > 0

(The cluster is actively fixing itself). 2. Write Failure Signal:

rate(starrocks_be_engine_requests_total{type="load", status="failed"}[5m]) > 0

(Writes are failing, likely due to quorum issues). 3. The "Node Down" Trigger:

count(starrocks_be_node_status == 0) > 0

(One or more BEs are down, which is the most common cause of a Yellow state). If you need the exact number of unhealthy tablets in Prometheus, you will need to use a tool like the

sql_exporter

to scrape the

information_schema.fe_metrics

table or the

SHOW PROC

output. References • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/resource_management/Replica.md References • administration/management/monitoring/metrics.md