@Rocky we deployed a shared-data cluster and I want to know the right metrics to check in prometheus and grafana regarding the status for fe and cn nodes. Can you point me to the correct docs?

For monitoring a StarRocks shared-data cluster (compute-node based), you should focus on the specific metrics designed for storage-compute separation, as well as general node health. 1. Key Documentation The most relevant documentation pages for your setup are: * Monitoring Metrics for Shared-data Clusters: This is the primary guide for shared-data specific metrics. * Monitor and Alert with Prometheus and Grafana: General setup for scraping metrics from FE and CN nodes. * General Monitoring Metrics: Core metrics for query performance, CPU, and memory. ────────── 2. Recommended Dashboards StarRocks provides two specialized Grafana dashboard templates for shared-data clusters. You can usually find these in the StarRocks GitHub repository under the

monitor

directory or via the official website: 1. Shared-data Dashboard: Focuses on high-level shared-data operations. * Publish Version: Latency and QPS for version publishing (critical for data visibility). * Metadata: Performance of

Get/Put Tablet Metadata

and

Txn Log

operations. * Vacuum: Status of background garbage collection (cleaning up old versions in object storage). 2. Starlet Dashboard: Focuses on the low-level I/O between CN nodes and Object Storage (S3/GCS/Azure). * S3/Object Storage IO:

fslib_s3_single_upload_latency

,

fslib_read_throughput

. * Cache Performance:

fslib_cache_hit_ratio

(very important for CN performance). ────────── 3. Critical Metrics to Watch For FE Nodes (Control Plane) FE nodes in shared-data mode manage metadata and coordinate transactions with object storage. *

starrocks_fe_publish_version_daemon_loop_total

: Ensures the background version publisher is running. * Metadata Latency: Monitor the latency of

GetTabletMeta

and

PutTabletMeta

to ensure the metadata service isn't a bottleneck. * Query Latency/QPS: Standard metrics like

starrocks_fe_query_latency_ms

and

starrocks_fe_query_err_rate

. For CN Nodes (Compute Plane) CN nodes handle the computation and data caching. * Data Cache Usage: *

datacache_mem_used_bytes

/ `datacache_disk_used_bytes`: Check how much local cache is being utilized. * `datacache_hit_ratio`: High hit rates are essential for performance in shared-data mode. * I/O Metrics: * `fslib_read_io_latency`: Latency of reads from object storage. * `fslib_read_throughput`: Throughput of data being pulled from the cloud bucket. * System Health: * `be_cpu_selftest`: General CPU usage. * `be_memory_usage_bytes`: CN memory consumption. 4. Endpoints To verify metrics are being exported, you can curl the endpoints directly: * FE Metrics:

http://<FE_IP>:<FE_HTTP_PORT>/metrics

(Default port: 8030) * CN Metrics:

http://<CN_IP>:<CN_HTTP_PORT>/metrics

(Default port: 8040) Note: For detailed table-level or user-level metrics, you may need to add parameters like

?with_table_metrics=all

and provide Basic Auth (ADMIN user) as described in the Monitor and Alert doc. References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md

@Rocky What if I want to get the following? 1. Number of Active and Inactive FE 2. Number of Active and Inactive CN 3. CPU and memory usage for FE 4. CPU and memory usage for CN

To get the information you're looking for regarding node status and resource usage in StarRocks, you can use several SQL commands and system views. 1. Number of Active and Inactive nodes (FE and CN) You can retrieve the status of all nodes using the

SHOW

commands. To get the specific counts, you would typically look at the

Alive

column. * For FEs:

sql
    SHOW FRONTENDS;

* Active: Rows where

Alive

is

true

. * Inactive: Rows where

Alive

is

false

. * For CNs:

sql
    SHOW COMPUTE NODES;
    -- Or use the PROC path
    SHOW PROC '/compute_nodes';

* Active: Rows where

Alive

is

true

. * Inactive: Rows where

Alive

is

false

. 2. CPU and Memory usage for CN The

SHOW COMPUTE NODES

command provides real-time resource usage directly in the output:

sql
SHOW COMPUTE NODES;

Look for these columns in the result: * CpuUsedPct: Current CPU usage percentage. * MemUsedPct: Current memory usage percentage. 3. CPU and Memory usage for FE Unlike CNs and BEs, the

SHOW FRONTENDS

command does not display CPU and memory usage directly. Since the FE is a Java-based process, resource monitoring is typically done via the following methods: * System Metrics Table: You can query the

information_schema.fe_metrics

table for specific JVM or system metrics if they are enabled.

sql
    SELECT * FROM information_schema.fe_metrics
    WHERE NAME LIKE '%jvm_heap_byte_used%'
       OR NAME LIKE '%cpu%';

* Metrics Endpoint: You can access the metrics in Prometheus format by visiting:

http://<FE_IP>:<FE_HTTP_PORT>/metrics

Look for: *

jvm_heap_size_bytes

(Memory) *

system_cpu_usage

(CPU) * FE Web UI: You can also view basic node health and status by logging into the FE HTTP interface (default port

8030

). Summary Table of Commands Requirement | SQL Command / Source | Key Column(s) FE Status |

SHOW FRONTENDS

|

Alive

CN Status |

SHOW COMPUTE NODES

|

Alive

CN CPU/Mem |

SHOW COMPUTE NODES

|

CpuUsedPct

,

MemUsedPct

FE CPU/Mem |

information_schema.fe_metrics

|

VALUE

(where Name matches metric) Rocky's Tip: If you are running a production cluster, I highly recommend setting up the StarRocks Grafana Dashboard. It aggregates all these metrics (CPU, Memory, RPC Latency, etc.) into a single visual view for both FE and CN/BE nodes! References • sql-reference/information_schema/fe_metrics.md • administration/management/monitoring/Monitor_and_Alert.md • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • sql-reference/sys/sys.md

@Rocky what I want is to build the dashboard in grafana with prometheus as the data source. Right now, i'm thinking of having 3 sections 1. Starrocks Cluster Overview 2. FE Monitoring 3. CN Monitoring Can you suggest me a minimalist sets of visuals that will make monitoring the starrocks cluster in shared data mode easier and relevant?

Hello! I'm Rocky. Building a dashboard for a Shared Data (compute-storage separation) cluster is a great way to ensure your performance and costs are optimized. Since you want a minimalist setup, you should focus on the "Critical Path" of a query: from the FE (Metadata/Parsing) to the CN (Computation/Cache) to the Object Storage. Here is a suggested minimalist set of visuals for your Grafana dashboard: 1. StarRocks Cluster Overview Purpose: A "Heartbeat" view to see if the cluster is healthy at a glance. * Node Status (Gauge): Number of Alive FEs vs. Alive CNs. * Query Throughput (Graph): Total QPS (Queries Per Second). _ _Metric:*

irate(starrocks_fe_query_total[1m])

* Query Latency (Graph): P99 and P95 latency. In shared-data mode, this helps identify if object storage latency is creeping in. _ _Metric:*

starrocks_fe_query_latency_ms

* Error Rate (Bar/Graph): Number of failed queries to catch syntax errors or resource exhaustion quickly. 2. FE Monitoring Purpose: Monitor the "Brain" of the cluster and its metadata health. * JVM Heap Usage (Graph): Shared-data FEs handle heavy metadata and planning. If the heap hits >80% consistently, you might see "Stop the World" pauses. * Active Connections (Gauge): Number of current client sessions. _ _Metric:*

starrocks_fe_connection_total

* Edit Log Write Latency (Graph): Vital for shared-data. High latency here means the Leader FE is struggling to persist metadata changes. _ _Metric:*

starrocks_fe_edit_log_write_latency_ms

* Transaction Status (Graph): Commits vs. Aborts. This monitors the health of your ingestion (Stream Load, Routine Load). 3. CN Monitoring (Shared-Data Focused) Purpose: Monitor compute efficiency and the "Local Cache" performance. * CPU & Memory Utilization (Graph): Per-node resource usage to see if you need to scale your CN group. * Data Cache Hit Rate (Gauge): The most important metric for Shared Data. This tells you how much data is being read from the local SSD cache vs. being pulled from S3/GCS/Azure. _ _Metric:*

starrocks_be_data_cache_read_block_cache_hit_ratio

(often found under the Starlet metrics). * Object Storage IO Latency (Graph): Average time to "GET" or "PUT" objects from your bucket. If this spikes, your query performance will drop regardless of CN power. _ _Metric:*

starlet_s3_get_object_latency_ms

* Scan Throughput (Graph): Rows/Bytes scanned per second. This helps distinguish between heavy "analytical" queries and "point" lookups. Pro-Tip for Shared-Data: StarRocks provides official Grafana JSON templates specifically for Shared-Data clusters. I recommend downloading the "Dashboard template for Shared-data Cluster - General" from our documentation. You can import it into Grafana and then delete the "noise" to match this minimalist list! Would you like the specific Prometheus expressions for any of these visuals? References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/Monitor_and_Alert.md • quick_start/shared-data.md

@Rocky i have this included in the metrics

increase(starrocks_fe_query_analysis_err_total[5m]) + increase(starrocks_fe_query_internal_err_total[5m])

it is showing 1.25..How can i investigate the actual error?

To investigate the errors indicated by your metrics (

starrocks_fe_query_analysis_err_total

and

starrocks_fe_query_internal_err_total

), you should look into the FE (Frontend) logs. The value of

1.25

from an

increase()

function usually indicates that a small number of errors occurred during that 5-minute window. Here is the step-by-step process to find the actual error messages: 1. Check

fe.audit.log

(Primary Source) The audit log records every SQL statement executed, its status, and any associated error codes. This is the best place to identify which specific queries are failing. * Location:

${STARROCKS_HOME}/fe/log/fe.audit.log

* What to look for: * Search for entries where the

state

is

ERR

. * Check the

errorCode

column for the specific error message. * Check the

stmt

column to see the actual SQL that caused the error. If you have the AuditLoader plugin installed: You can query the errors directly using SQL:

sql
SELECT timestamp, queryId, user, errorCode, stmt 
FROM starrocks_audit_db__.starrocks_audit_tbl__ 
WHERE state = 'ERR' 
ORDER BY timestamp DESC 
LIMIT 10;

2. Differentiate the Error Types * Analysis Errors (

starrocks_fe_query_analysis_err_total

): These are usually SQL syntax errors or semantic errors (e.g., referencing a table or column that doesn't exist, type mismatches). _ _Investigation:* The

fe.audit.log

will show the specific syntax error returned to the client. * Internal Errors (

starrocks_fe_query_internal_err_total

): These indicate unexpected system failures during query processing, such as a NullPointerException, resource exhaustion, or a bug in the FE's optimizer/executor. _ _Investigation:* Check

fe.warn.log

or

fe.log

for Java stack traces. Search for the

queryId

found in the audit log to correlate the logs. 3. Check

fe.warn.log

and

fe.log

If the audit log doesn't provide enough detail (especially for Internal Errors), the system logs will contain the full Java stack trace. * Location:

${STARROCKS_HOME}/fe/log/fe.warn.log

* Action: Search for keywords like

Exception

,

RuntimeException

, or specific error strings found in the audit log. 4. Use Query Dump (For complex Internal Errors) If you encounter a persistent internal error that is hard to diagnose, you can enable the query dump feature to capture the complete context (metadata, statistics, and plan) of the failed query:

sql
SET enable_query_dump = true;

The details will be saved in

${STARROCKS_HOME}/fe/log/fe.dump.log

. Summary Recommendation Start by searching your

fe.audit.log

for

|ERR|

. This will tell you who ran what SQL and what the error message was. If it's an "Internal Error," take the

QueryId

from that log line and search for it in

fe.warn.log

to see the full stack trace. References • administration/management/logs.md • administration/management/audit_loader.md • faq/operation_maintenance_faq.md

@Rocky is there a better way to avoid always checking the .log files?

Checking

.log

files manually can be tedious. Depending on what you are looking for in those logs, StarRocks provides several automated and SQL-driven alternatives to monitor and troubleshoot your cluster. 1. For Query History: Use AuditLoader If you find yourself constantly checking

fe.audit.log

to see who ran what query or why a query was slow, you should use the AuditLoader plugin. * What it does: It automatically reads the audit logs and loads them into a StarRocks table (e.g.,

starrocks_audit_db*.starrocks_audit_tbl*

). * The Benefit: You can search, filter, and analyze query history using standard SQL. * How to set it up: You create a target table and install the plugin via the

FE/plugins

directory. 2. For Health & Performance: Prometheus and Grafana Instead of looking at

fe.log

or

<http://be.INFO|be.INFO>

for errors or performance spikes, the standard practice is to use the Prometheus + Grafana stack. * Metrics: StarRocks exposes hundreds of metrics (at

http://<FE_IP>:8030/metrics

and

http://<BE_IP>:8040/metrics

). * Dashboards: You can use pre-built Grafana dashboards to visualize CPU, memory, IO, query latency, and connection counts. * Alerting: You can set up alerts to notify you (via Slack, Email, PagerDuty) when a node goes down or memory usage is too high, so you don't have to "check" manually. 3. For Real-time Status: System Tables StarRocks provides built-in system tables in the

information_schema

database that allow you to check the state of the cluster via SQL: *

information_schema.be_metrics

/ `fe_metrics`: Query real-time metrics for each node. * `information_schema.be_tablets`: Check for tablet health and distribution issues. *

SHOW BACKENDS

/ `SHOW FRONTENDS`: The

ErrMsg

column in these commands often provides the exact reason a node is down or why heartbeats are failing, which is often what people look for in logs. 4. For Query Debugging: Query Profiles If you are checking logs to debug query performance: * Use

SET enable_profile = true;

and then

get_query_profile('<query_id>');

. * This provides a detailed execution breakdown that is much more informative than the standard logs. 5. For Centralized Logging (Advanced) If you still need the raw system logs (like

fe.log

or

be.WARNING

) for deep troubleshooting but want to avoid the CLI, the industry standard is to use a logging agent (like Vector, Filebeat, or Fluentd) to ship these logs to a centralized log management system (like an ELK stack or Loki). This allows you to search across all nodes from a single UI. Summary Recommendation: * Start with AuditLoader for query history. * Set up Prometheus/Grafana for health monitoring. * Use

SHOW BACKENDS

and

information_schema

for quick status checks. References • sql-reference/information_schema/be_metrics.md • sql-reference/information_schema/be_bvars.md • sql-reference/information_schema/fe_metrics.md • sql-reference/information_schema/be_tablets.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_BACKENDS.md • administration/management/BE_configuration.md