@Rocky what are new prom metrics StarRocks got since 4.0.0?

Since version 4.0.0, StarRocks has introduced several new Prometheus metrics and enhanced existing ones to improve observability, particularly around transaction processing, data caching, and the new merge commit feature. 1. FE Transaction Latency Metrics In version 4.0, StarRocks introduced a set of

summary

-type metrics on the FE to provide high-resolution latency distributions (quantiles: 0.75, 0.95, 0.98, 0.99, 0.999) for transaction phases. These are reported exclusively by the Leader FE. * `starrocks_fe_txn_total_latency_ms`: Total end-to-end latency (prepare to finish). * `starrocks_fe_txn_write_latency_ms`: Latency of the write phase (prepare to commit). * `starrocks_fe_txn_publish_latency_ms`: Latency of the publish phase (commit to finish). * `starrocks_fe_txn_publish_schedule_latency_ms`: Time spent waiting in the publish queue. * `starrocks_fe_txn_publish_execute_latency_ms`: Active execution time of the publish task. * `starrocks_fe_txn_publish_ack_latency_ms`: Final acknowledgment latency. * `starrocks_fe_publish_version_daemon_loop_total`: Total runs of the publish-version daemon. 2. Merge Commit Metrics (BE) Introduced to monitor the "Merge Commit" feature, which optimizes high-concurrency small-batch stream loads. * Throughput/Status:

merge_commit_request_total

,

merge_commit_request_bytes

,

merge_commit_success_total

,

merge_commit_fail_total

. * Queueing:

merge_commit_pending_total

,

merge_commit_pending_bytes

. * Internal Latency (reported in microseconds):

merge_commit_request

(end-to-end),

merge_commit_pending

,

merge_commit_wait_plan

,

merge_commit_append_pipe

,

merge_commit_wait_finish

. 3. Enhanced DataCache Metrics (BE) As of 4.0, DataCache configurations were simplified, and monitoring was unified. Version 4.0.3 further added specific statistics for native tables. *

datacache_mem_quota_bytes

/ `datacache_mem_used_bytes`: Memory limit and current usage. *

datacache_disk_quota_bytes

/ `datacache_disk_used_bytes`: Disk limit and current usage. * `datacache_meta_used_bytes`: Memory used for cache metadata. *

block_cache_hit_bytes

/ `block_cache_miss_bytes`: Cache statistics (initially for external tables, but v4.0.3 enhanced hit rate stats for native tables as well). 4. Query Queue and Pipe Metrics * `starrocks_fe_query_queue_pending`: Number of queries currently in the queue. * `starrocks_fe_query_queue_total`: Historical total of queued queries. * `starrocks_fe_query_queue_timeout`: Queries that timed out while waiting in the queue. * `starrocks_be_pipe_prepare_pool_queue_len`: Task queue length for the pipeline prepare thread pool. 5. Metric Label Changes *

is_leader

label: Added to several FE metrics (e.g., transaction and query metrics) to distinguish the Leader FE from Followers. *

type

label: Added to transaction metrics to categorize by load source (e.g.,

stream_load

,

routine_load

). For a complete and up-to-date list of all metrics, you can refer to the General Monitoring Metrics documentation. References • sql-reference/System_variable.md • administration/management/resource_management/spill_to_disk.md • administration/management/BE_configuration.md • administration/management/monitoring/metrics.md

@Rocky what are metrics about table lock?

StarRocks monitors metadata locking (which includes database and table locks) primarily through the Frontend (FE). Below are the key metrics and observability tools for tracking table-level and database-level locks. 1. Prometheus Metrics (Slow Locks) Since version 3.2.0, StarRocks provides high-resolution histograms to monitor "slow locks"—locks that are held or waited for longer than a specific threshold (controlled by the FE configuration

slow_lock_threshold_ms

, default is

3000

ms). *

starrocks_fe_slow_lock_held_time_ms

: * Description: Tracks the maximum duration a lock was held by its owners once a slow lock event is detected. * Usage: Useful for identifying long-running operations (like large metadata updates or heavy schema changes) that are blocking other requests. *

starrocks_fe_slow_lock_wait_time_ms

: * Description: Tracks how long threads were stuck waiting to acquire a lock. * Usage: The primary indicator of lock contention. High values suggest that concurrent operations (e.g., many simultaneous loads and queries) are fighting for the same database/table lock. 2. Real-time Monitoring:

fe_locks

System Table For a real-time view of what is currently locked, you can query the system schema. This provides more granular detail than Prometheus, including the specific table or database involved.

sql
SELECT * FROM information_schema.fe_locks;
-- or
SELECT * FROM sys.fe_locks;

Key columns in this table: * `lock_type`: The type of object (e.g.,

DATABASE

). * `lock_object`: The name of the database or table. * `lock_mode`:

EXCLUSIVE

(usually for DDL/Writes) or

SHARED

(usually for Reads/Queries). * `hold_time_ms`: How long the lock has been held so far. * `granted`: Whether the thread actually has the lock or is still waiting. * `waiter_list`: A list of other threads blocked by this lock. 3. Transaction & Publish Metrics Table locks are frequently held during the Publish Version phase of a transaction (when data becomes visible). If you see high lock contention, check these metrics: *

starrocks_fe_txn_publish_latency_ms

: The time taken to make data visible after a commit. *

starrocks_fe_txn_publish_schedule_latency_ms

: Time spent in the queue waiting for the publish task (often due to waiting for a metadata lock). 4. Deadlock Detection StarRocks includes a DeadLockChecker (available in versions 2.5.15+, 3.0.9+, 3.1.6+, and 3.2.0+). When a potential deadlock or severe lock contention is detected, it logs detailed JSON information to

fe.log

. This log includes: * The

lockDbName

and

lockState

. * A

dumpThread

stack trace of the lock owner. * A list of

lockWaiters

. Summary of Relevant Configurations Configuration | Default | Description

slow_lock_threshold_ms

| 3000 | Threshold for a lock to be recorded in Prometheus metrics and logs.

enable_deadlock_checker

| true | Enables the background thread that detects and logs metadata deadlocks. References • administration/management/monitoring/metrics.md • knowledge/trouble_shooting/fe_deadlock_problems.md • knowledge/usage/best-practice.md

@Rocky give me correct full names for "*Enhanced DataCache Metrics (BE)" for prom metrics with brief Title I can use for chart name and more detailed description*

In StarRocks 4.0 (specifically enhanced in 4.0.3 to support native tables), the DataCache (formerly known primarily as Block Cache) metrics provide a unified view of the caching performance across memory and disk. The "correct full names" exported by the Backend (BE) to Prometheus include the

starrocks_be_

prefix. Enhanced DataCache Capacity & Usage These metrics monitor the footprint and limits of the cache tiers. Full Prometheus Metric Name | Chart Title | Detailed Description

starrocks_be_datacache_mem_quota_bytes

| DataCache RAM Limit | The total memory (RAM) capacity allocated for the memory-tier cache.

starrocks_be_datacache_mem_used_bytes

| DataCache RAM Usage | The actual bytes of memory currently occupied by cached data blocks.

starrocks_be_datacache_disk_quota_bytes

| DataCache Disk Limit | The total disk space allocated for the persistent-tier cache across all paths.

starrocks_be_datacache_disk_used_bytes

| DataCache Disk Usage | The actual bytes of disk space consumed by cached data blocks.

starrocks_be_datacache_meta_used_bytes

| DataCache Metadata Mem | Memory used for the cache index and metadata; high usage may indicate too many small files/blocks. Enhanced DataCache Performance (Efficiency) These metrics track how effectively the cache is serving requests. Note that "Block Cache" naming is retained for throughput metrics as it refers to the underlying block-level storage engine. Full Prometheus Metric Name | Chart Title | Detailed Description

starrocks_be_block_cache_hit_bytes

| DataCache Hit Throughput | Cumulative bytes successfully read from the cache (memory or disk), avoiding remote I/O.

starrocks_be_block_cache_miss_bytes

| DataCache Miss Throughput | Cumulative bytes that were not found in the cache and had to be fetched from the backend storage.

starrocks_be_block_cache_read_disk_bytes

| DataCache Disk Read BW | Throughput of data read specifically from the disk-tier of the cache.

starrocks_be_block_cache_write_disk_bytes

| DataCache Disk Write BW | Throughput of new data being written into the disk-tier cache. Recommended Chart Calculation To visualize the Cache Hit Rate (By Volume), use the following PromQL in your chart:

promql
sum(irate(starrocks_be_block_cache_hit_bytes[5m])) / 
(sum(irate(starrocks_be_block_cache_hit_bytes[5m])) + sum(irate(starrocks_be_block_cache_miss_bytes[5m])))

*Note: Since version 4.0.3, these metrics include statistics for Native Tables if the DataCache is enabled for them, whereas previously they were primarily for External Tables (Iceberg, Hudi, etc.).* References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md

@Rocky any metrics for page cache?

While the DataCache (Block Cache) is used primarily for external tables and shared-data mode, StarRocks also provides specific metrics for the Storage Page Cache, which caches data pages, index pages, and dictionary pages for native tables (OLAP) to speed up local disk I/O. In Prometheus, these metrics are exported by the Backend (BE) with the

starrocks_be_

prefix. Page Cache Capacity & Usage These metrics track the memory footprint of the native storage page cache. Full Prometheus Metric Name | Chart Title | Detailed Description

starrocks_be_page_cache_mem_quota_bytes

| Page Cache Mem Limit | The maximum memory allocated for the storage page cache (configured via

storage_page_cache_limit

).

starrocks_be_page_cache_mem_used_bytes

| Page Cache Mem Usage | The current amount of memory occupied by cached native data/index pages. Page Cache Performance (Efficiency) These metrics are used to calculate the efficiency of the cache for local storage. Full Prometheus Metric Name | Chart Title | Detailed Description

starrocks_be_page_cache_hit_count

| Page Cache Hit Count | Cumulative number of times a requested data/index page was found in the page cache.

starrocks_be_page_cache_miss_count

| Page Cache Miss Count | Cumulative number of times a requested page was not in memory and required a local disk read. Recommended Chart Calculation To visualize the Page Cache Hit Rate (by request count), use the following PromQL:

promql
sum(irate(starrocks_be_page_cache_hit_count[5m])) / 
(sum(irate(starrocks_be_page_cache_hit_count[5m])) + sum(irate(starrocks_be_page_cache_miss_count[5m])))

Difference between Page Cache and DataCache * Page Cache (

starrocks_be_page_cache_...

): Caches internal segment file pages for native OLAP tables. Managed by the storage engine. * DataCache (

starrocks_be_block_cache_...

): Caches data from remote storage (S3/HDFS) for External Tables or Shared-data clusters. This is the newer, more unified caching layer. _Note: In some versions/configurations, the

page_cache_mem_used_rate

and

page_cache_hit_rate

are available via the internal BE HTTP API (

/metrics

), but for Prometheus/Grafana, it is best practice to use the raw counters (

hit_count

,

miss_count

) to calculate rates over time._ References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md • administration/management/BE_configuration.md • administration/management/monitoring/monitor_and_alert.md References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md

@Rocky I see

starrocks_be_page_cache_hit_count

but not miss count, is it different name? I use version 4.1.4

In StarRocks version 4.1.4, the Prometheus metric for "misses" in the native storage Page Cache is indeed named differently than you might expect. Instead of an explicit "miss" counter, StarRocks provides a "lookup" counter. To determine the miss count and hit rate, use the following metrics: The Metric Names *

starrocks_be_page_cache_hit_count

: The cumulative number of successful cache hits. *

starrocks_be_page_cache_lookup_count

: The cumulative number of total attempts (Lookups). _ _This is the "Total" counter (Hits + Misses).* How to calculate "Misses" If you want to visualize the number of misses in Grafana, you must calculate the difference between lookups and hits:

promql
# Page Cache Misses (per second)
irate(starrocks_be_page_cache_lookup_count[5m]) - irate(starrocks_be_page_cache_hit_count[5m])

How to calculate "Hit Rate" The hit rate is calculated by dividing hits by lookups:

promql
# Page Cache Hit Rate %
sum(irate(starrocks_be_page_cache_hit_count[5m])) / sum(irate(starrocks_be_page_cache_lookup_count[5m]))

Why the confusion? * Page Cache (Native Tables): Uses

hit_count

and

lookup_count

. This cache stores data pages for local OLAP tables. * DataCache / Block Cache (External/Shared-Data): Uses

block_cache_hit_count

and

block_cache_miss_count

. This is the newer, unified caching layer for remote data. If you are monitoring native table performance, stick with the

_lookup_count

and

_hit_count

metrics. References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md References • administration/management/monitoring/metrics.md • data_source/data_cache_observe.md