@Rocky whats this metric is in shared architechture fslib_read_io_latency_count?

In StarRocks shared-data (compute-storage decoupled) architecture,

fslib_read_io_latency_count

is a monitoring metric that represents the total number of read I/O operations performed by the Starlet storage library (fslib). Key Details: * Metric Type: It is the

_count

component of a Prometheus Histogram metric (

fslib_read_io_latency

). * Meaning: It tracks how many read requests have been issued to the underlying storage (typically S3/object storage or the local Data Cache). * Use Case: While the

_sum

and

_bucket

components of this metric are used to calculate average and quantile latencies (P99, P95, etc.), the

_count

component specifically tells you the total volume of read requests over time. Context in Shared-Data Architecture: In a shared-data cluster, StarRocks uses a storage abstraction layer (Starlet/fslib) to handle data. This metric covers: 1. S3/Object Storage Reads: When data is fetched directly from the remote object store. 2. Data Cache Reads: In many configurations, this metric also reflects reads from the local disk-based Data Cache (StarCache) when the data is already cached. If you are looking at this in a Grafana dashboard, it is often used to calculate Read IOPS by using the

rate()

function in PromQL:

rate(fslib_read_io_latency_count[5m])

References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • using_starrocks/caching/block_cache.md

@Rocky yes but why it shows nothing whilst I see fslib_list_dir?

In StarRocks shared-data architecture, seeing

fslib_list_dir

(or metadata operations) while

fslib_read_io_latency_count

remains empty or zero is usually due to one of the following reasons: 1. Metadata vs. Data Operations The most common reason is that you are performing metadata-only operations. *

fslib_list_dir

: This increments when the system lists directories in your object storage (S3/GCS/MinIO) to check for file existence, metadata updates, or during background heartbeats. This happens even if no user queries are running. *

fslib_read_io_latency_count

: This only increments when actual data blocks (the

.cols

or

.segment

files) are being read from the storage layer. If you haven't executed a

SELECT

query that requires scanning data, this counter will stay at zero. 2. High Cache Hit Ratio (Data Cache) If Data Cache is enabled and the data you are querying is already "hot" in the local cache: * In some versions of StarRocks, the

fslib_read_io_latency

metric specifically tracks Remote I/O (S3/Object Storage). * If your query is satisfied entirely by the local disk cache (StarCache) or memory (Page Cache), it may skip the remote I/O path that triggers this specific counter. * Check the

fslib_cache_hits

or

fslib_cache_hit_ratio

metrics to see if the data is being served from the cache instead. 3. Metric Prefix Discrepancy Depending on how you are scraping metrics, ensure you are looking for the full name. StarRocks BE metrics are typically exported with a prefix: * Check for

starrocks_be_fslib_read_io_latency_count

. * If you are using a Grafana dashboard, verify that the PromQL isn't filtering by a specific label (like

storage_type="s3"

) that might be empty if you are using a different storage provider (like HDFS or MinIO). 4. Query Execution Profile To confirm if the system should be reading data, run a query and check the Query Profile: 1. Run

SET enable_profile = true;

. 2. Run your query. 3. Check the profile (via

SHOW PROFILE

or the FE UI). 4. Look for

RemoteScan

metrics. If

WaitStorageIOTime

is 0 and

ReadBytes

is 0, it means no I/O was triggered at the fslib level for that query. Summary Recommendation: Try running a "cold" query (one that hasn't been run before, or after clearing the cache) that scans a large table. If

fslib_read_io_latency_count

still shows nothing, it is likely a version-specific naming issue or a scraping configuration error. References • administration/management/monitoring/metrics-shared-data.md • using_starrocks/caching/block_cache.md • administration/management/monitoring/metrics.md References • administration/management/monitoring/metrics-shared-data.md • deployment/shared_data/feature-support-shared-data.md • faq/shared_data_faq.md • deployment/shared_data/minio.md • using_starrocks/caching/block_cache.md • deployment/shared_data/gcs.md