@Rocky in shared architechture fslib_read_io_latency_count this is the read latency when query is running on top of external catalog which queries data from s3? or its just about starrocks meta on object store?

The metric

fslib_read_io_latency_count

(and the related

fslib_read_io_latency

histogram) primarily measures the read latency of StarRocks native table data stored on object storage (like S3) in a shared-data (cloud-native) architecture. Here is the breakdown of what it covers: 1. Native Data Segments (Primary Use) In StarRocks' shared-data architecture, data is stored as segments on S3. The

fslib

(FileSystem Library) is part of the Starlet storage engine, which handles I/O for these native tables. This metric tracks the actual latency of the I/O calls to S3 when StarRocks reads these data segments for queries. 2. Is it for External Catalogs? Usually, no. * External Catalog queries (e.g., querying Iceberg, Hive, or Hudi on S3) typically use a different scanner path and different metrics, such as

starrocks_be_files_scan_num_bytes_read

or

starrocks_be_files_scan_num_files_read

. * However, if you have Data Cache enabled for your external catalogs in a shared-data cluster, the reads that miss the cache and fetch data from S3 may be tracked via Starlet's I/O layer, but the primary metrics for external catalog performance are found under the

be_files_scan

or

be_scan

categories. 3. Is it just about Metadata? No. * Data vs. Meta: This metric is for data I/O. There is a separate metric called

filesystem_meta_operations

for metadata-related operations like directory listing or file creation on S3. * StarRocks Metadata: The primary cluster metadata (table schemas, etc.) is managed by the Frontend (FE) and is not stored on S3 in a way that would be tracked by this BE/CN-level I/O metric. This metric might include reading low-level storage metadata (like segment footers) stored on S3, but its main volume comes from reading actual table data rows/columns. Summary * Target: Native StarRocks tables in Shared-data mode. * Data Type: Actual data segments (and some segment-level metadata). * External Catalogs: Generally not covered by this metric; look for

starrocks_be_files_scan_*

metrics for external data lake queries on external catalogs. References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md

@Rocky whats starlet and fslib?

In the context of StarRocks' shared-data (cloud-native) architecture, Starlet and fslib are internal components responsible for managing how data is stored and accessed on object storage. 1. Starlet Starlet is the internal storage engine and resource management layer specifically designed for StarRocks' shared-data architecture. * Role: It acts as the "brain" for storage on a Compute Node (CN) or Backend (BE). While the standard StarRocks storage engine handles local disks, Starlet manages the lifecycle of data stored remotely (on S3, GCS, etc.). * Functions: It handles data consistency, metadata caching for remote files, and coordinates with the Frontend (FE) to manage which data segments belong to which tablets in the cloud-native mode. 2. fslib (FileSystem Library) fslib is an internal I/O abstraction library within Starlet. * Role: It provides a unified interface for StarRocks to perform file operations (read, write, list, delete) across different storage backends. * Functions: * Backend Abstraction: It hides the complexity of whether the data is on AWS S3, Google Cloud Storage, Azure Blob, or HDFS. * I/O Optimization: It handles features like asynchronous I/O, prefetching, and Data Caching (managing the local disk cache for remote data). * Metrics: This is why the metrics you see (like

fslib_read_io_latency

) are prefixed with `fslib`; they are captured at this abstraction layer to measure the actual performance of the storage subsystem. Summary Relationship * Starlet is the module (the "what" and "where" of cloud storage). * fslib is the I/O tool (the "how" of reading/writing those files). When you see

fslib_read_io_latency_count

, you are looking at the number of read operations handled by this library, which typically corresponds to reading native StarRocks data segments from your remote object store. References • introduction/Architecture.md • quick_start/shared-nothing.md • administration/management/monitoring/metrics-shared-data.md

@Rocky what should I look into if I wanna investigate latency on s3 on shared data reading (not native starrocks tablet just some iceberg detallake tables on external catalogs)

To investigate S3 read latency in a StarRocks shared-data architecture, you should look at three main areas: Prometheus Metrics, the Query Profile, and Storage Configuration. 1. Key Prometheus Metrics StarRocks provides specific metrics for the Starlet/fslib layer. If you have the StarRocks Grafana dashboard, look for the "Starlet" section. *

fslib_read_io_latency

(Quantile/Average): This is the direct measure of how long it takes to read from S3. _ _What to look for:* High

P99

latency. S3 typically has a latency of 10-100ms. If you see spikes into the seconds, it indicates S3 throttling or network congestion. * `fslib_cache_hit_ratio`: Since shared-data clusters use a local disk cache (Data Cache), S3 is only hit when there is a cache miss. _ _What to look for:* A low hit ratio means most of your query latency is subject to S3's remote I/O speed. * `fslib_read_iosize`: _ _What to look for:* S3 performs better with larger read sizes. If your

iosize

is consistently very small (e.g., < 64KB), you may be suffering from "death by a thousand cuts" (high overhead per request). *

fslib_read_throughput

& `fslib_read_iops`: Check if you are hitting the throughput limits of your network or the IOPS limits of your S3 bucket. 2. Query Profile Analysis For a specific slow query, generate a profile (

SET enable_profile = true;

) and examine the

OLAP_SCAN_NODE

. *

DataCacheReadBytes

vs. `DataCacheWriteBytes`: * `DataCacheReadBytes`: Data read from local disk/memory cache. * `DataCacheWriteBytes`: Data fetched from S3 and then written to cache. * If

DataCacheWriteBytes

is high, the query is actively waiting on S3. * `IOTaskWaitTime`: Time spent waiting in the thread pool queue before the I/O actually starts. High values suggest the BE/CN is overloaded with too many concurrent I/O requests. * `ScanTime`: Total time spent in the scan operator. In shared-data mode, this is heavily influenced by the remote read latency. 3. Investigation Checklist 1. Check for S3 Throttling: If

fslib_read_io_latency

is high, check your S3 provider (e.g., AWS CloudWatch) for

SlowDown

errors or throttling. You may need to spread data across more prefixes. 2. Verify Data Cache Health: Use

SHOW BACKEND\COMPUTE NODE

to ensure the

DataCache

is

Enable: true

and that the disks assigned to the cache are not full or failing. 3. Network Bandwidth: Shared-data performance is often bound by the network throughput of the EC2/VM instances. Monitor the node-level network traffic to see if you are hitting the instance's bandwidth cap. 4. Warm-up: If this is a new cluster or new data, latency will be higher initially as the cache is cold. Use

CACHE SELECT...

to pre-warm the cache if necessary. Summary of Metrics to Query in Prometheus:

promql
# P99 Latency for S3 Reads
histogram_quantile(0.99, sum by (le) (rate(starrocks_be_fslib_read_io_latency_bucket[5m])))

# Cache Hit Ratio
rate(starrocks_be_fslib_cache_hits_total[5m]) / 
(rate(starrocks_be_fslib_cache_hits_total[5m]) + rate(starrocks_be_fslib_cache_misses_total[5m]))

References • administration/management/monitoring/metrics-shared-data.md • data_source/data_cache.md • best_practices/query_tuning/query_profile_operator_metrics.md References • best_practices/query_tuning/query_profile_operator_metrics.md • data_source/data_cache.md

@Rocky what should I look into if I wanna investigate latency on s3 on shared data reading (not native starrocks tablet just some iceberg detallake tables on external catalogs)

To investigate S3 read latency for external catalogs (like Apache Iceberg) specifically, the focus shifts from StarRocks' internal metadata to the Connector Scan path and the Data Cache layer. In StarRocks v3.4+, queries against external catalogs and native cloud-native tables share a unified Data Cache instance and I/O library (fslib/Starlet). Therefore, you should look into the following areas: 1. Prometheus Metrics (BE/CN Level) Even for external catalogs, the underlying S3 I/O is often routed through the

fslib

layer if Data Cache is enabled. Use these metrics to see raw S3 performance: *

starrocks_be_fslib_read_io_latency

: This histogram measures the actual round-trip time for S3 read requests. If the P99 is high (e.g., >200ms), S3 is likely the bottleneck (check for S3 throttling or high network latency). *

starrocks_be_block_cache_hit_bytes

vs.

starrocks_be_block_cache_miss_bytes

: For external catalogs, these metrics track whether the query is hitting the local disk cache or fetching from S3. High "miss" bytes directly translate to higher query latency due to S3 I/O. *

starrocks_be_fslib_read_iosize

: S3 performance is significantly better with larger read sizes. If your average

iosize

is small (e.g., <128KB), latency will feel much higher due to S3's request overhead. 2. Query Profile Analysis Run your query with

SET enable_profile = true;

and find the

CONNECTOR_SCAN_OPERATOR

(or

HdfsScanNode

) section. Key counters include: *

ScanTime

: The total time spent in the scan operator. *

IOTaskExecTime

: The actual time the threads spent performing I/O. If this is high, it confirms the storage backend (S3) is slow. *

IOTaskWaitTime

: Time spent waiting in the thread pool queue. If this is high, your BE/CN nodes are likely saturated with too many concurrent I/O requests. *

DataCacheReadBytes

: Bytes served from local disk/memory. *

DataCacheWriteBytes

: Bytes fetched from S3 to be cached. If this matches your total read volume, you have a cold cache problem. 3. Iceberg-Specific Metadata Latency If the "Planning" phase of your query is slow (before execution starts), the bottleneck might be fetching Iceberg metadata (manifest files) from S3: * Check FE Metrics: Look at

iceberg_metadata_cache_hit_ratio

. StarRocks caches Iceberg metadata on the Frontend. * Session Variables: Verify that

enable_iceberg_metadata_cache

is

true

. * Metadata Tables: You can query

<catalog>.<db>.<table>$files

to see how many files are being scanned. If there are thousands of small files, S3 latency will be compounded. 4. Checklist for S3 Performance * S3 Throttling: Look for

503 Slow Down

errors in your S3 provider's logs. This happens if you have too many concurrent requests on a single S3 prefix. * Instance Bandwidth: Check if your CN/BE nodes are hitting their network throughput limits (e.g., EC2 instance network caps). * Data Layout: Latency is often a symptom of "too many small files." Use Iceberg's

OPTIMIZE

or compaction to merge small Parquet files into larger ones (~128MB–256MB). References • administration/management/monitoring/metrics-shared-data.md • best_practices/query_tuning/query_profile_operator_metrics.md • data_source/iceberg_catalog.md • data_source/data_cache.md References • data_source/data_cache.md • data_source/data_cache_troubleshooting.md • administration/management/monitoring/metrics.md

@Rocky I use your grafana templates and which charts exactly should I look into to investigate latency between shared data (external catalog objects)

To investigate S3 latency for external catalog objects (Iceberg, Delta Lake, etc.) using the official StarRocks Grafana templates, you should focus on the Starlet Dashboard and the Data Cache section of the BE Dashboard. Since StarRocks v3.4.0, the Data Cache is unified, meaning external catalog I/O and shared-data native I/O often flow through the same internal library (fslib). 1. Starlet Dashboard (The most important for S3) This dashboard monitors the "Starlet" storage engine, which handles the actual communication with S3. * Section: FSLIB READ IO METRICS *

fslib read io_latency (quantile)

: This is the single most important chart. It shows the P99/P90 latency of reads from the storage backend (S3). If this is consistently high (>100ms), the bottleneck is S3 itself or the network. *

fslib read iosize (quantile)

: Check if your reads are too small. S3 performance degrades significantly if you are doing many small (e.g., <64KB) reads instead of larger ones. *

fslib read throughput

: Check if you are hitting the network bandwidth limit of your CN/BE nodes. * Section: FSLIB CACHE METRICS *

fslib cache hit ratio

: If this is low, every query is being forced to go to S3. A high hit ratio (e.g., >90%) means latency should be low regardless of S3 speed. * Section: S3 IO METRICS *

fslib s3 single/multi upload latency

: Useful if you are investigating write (Iceberg Sink/Insert) latency. 2. BE Dashboard (Data Cache / Block Cache) External catalog performance relies heavily on the Block Cache. * Section: Data Cache *

Block Cache Hit/Miss Bytes

: This chart specifically tracks bytes read from the local cache versus remote storage for external tables. Look for a high volume of

miss_bytes

during slow queries. *

DataCache Mem/Disk Used

: Ensure the cache isn't "full" and constantly evicting data, which leads to "thrashing" and high S3 latency as data is repeatedly re-fetched. 3. FE Dashboard (Metadata Latency) If the query planning is slow (the time before the first row starts moving), the bottleneck is likely the FE fetching Iceberg metadata from S3. * Section: Iceberg Metrics (if available in your template version) *

Iceberg Metadata Cache Hit Ratio

: High miss rates here mean the FE is waiting on S3 to read manifest files. *

iceberg_delete_duration

: Specifically for Iceberg v2 tables with position deletes. Summary Checklist for Latency: 1. Check

fslib read io_latency

(P99): Is S3 responding slowly? 2. Check `fslib cache hit ratio`: Are we missing the cache too often? 3. Check `fslib read iosize`: Are the files/reads too small for S3 to handle efficiently? Note: If you are using the older shared-nothing architecture with external tables, look specifically for the "Connector Scan" or "External Scan" panels in the BE dashboard. References • administration/management/monitoring/metrics-shared-data.md • using_starrocks/caching/block_cache.md • data_source/data_cache_observe.md • administration/management/monitoring/metrics.md References • administration/management/monitoring/metrics-shared-data.md • deployment/shared_data/minio.md • deployment/shared_data/gcs.md • deployment/shared_data/azure.md

@Rocky I see that fslib_read_io_latency_count{fstype="s3"} 3553 does not increase at all what can be the reason? (data definetely comes from s3 and there are too many reads)

If

fslib_read_io_latency_count{fstype="s3"}

is not increasing despite active S3 reads from an External Catalog (Iceberg/Delta Lake), it is usually due to one of the following reasons: 1. The Metric is Specific to Native Shared-Data Tables In StarRocks, the

fslib

(Starlet) metrics are primarily designed to monitor the StarRocks Shared-Data (Cloud-native) storage volume. * Native Shared-Data Tables: Use

fslib

to communicate with the S3 bucket defined in your

storage_volume

. * External Catalogs: While they also read from S3, they often use a different internal I/O path (the

Connector Scan

path). Even if you are running a shared-data cluster, reads for Iceberg or Delta Lake tables typically do not increment the Starlet/fslib S3 counters because they are not part of the Starlet-managed internal data volume. 2. You should look at "Block Cache" or "Data Cache" Metrics For External Catalogs, S3 read monitoring is moved to the Data Cache (also known as Block Cache) layer. Look for these metrics instead: *

starrocks_be_block_cache_miss_bytes

: This increases when StarRocks has to go to S3 because the data wasn't in the local cache. *

starrocks_be_block_cache_read_request_total

: The total number of requests to the cache system. *

starrocks_be_files_scan_num_bytes_read

: The total bytes read from external storage (S3/HDFS). 3. Check the Query Profile (The Source of Truth) To see exactly which metrics are being populated for your specific Iceberg/Delta Lake query, run:

sql
SET enable_profile = true;
-- Run your query
SHOW PROFILE FOR QUERY "your_query_id";

In the profile, look for the

CONNECTOR_SCAN_NODE

. You will find the following metrics which track S3 latency for external files: *

DataCacheWriteTimer

: This is the time spent fetching data from S3 to fill the cache. If your

fslib

metrics are zero, this timer will tell you the actual S3 latency. *

ScanTime

: Total time spent scanning. *

IOTaskExecTime

: The actual execution time of the I/O tasks. 4. StarRocks Version (Unified Cache) If you are on a version older than v3.4.0, the I/O paths for external catalogs and native tables were entirely separate. * From v3.4.0 onwards, StarRocks introduced a Unified Data Cache. In this version, you might start seeing some overlap, but generally, the

fstype="s3"

label in

fslib

still refers specifically to the backend storage of the StarRocks cluster itself, not the arbitrary S3 buckets used by external Iceberg catalogs. Recommendation: Switch your Grafana investigation to the "Data Cache" or "External Scan" panels. If you are using the official templates: 1. Open the BE Dashboard. 2. Find the Data Cache section. 3. Look for Block Cache Hit/Miss and Read/Write Bytes. 4. If those are also zero, verify that you have enabled the cache for your catalog via

PROPERTIES ("enable_remote_file_cache" = "true")

. References • administration/management/monitoring/metrics-shared-data.md • using_starrocks/caching/block_cache.md • data_source/data_cache.md • best_practices/query_tuning/query_profile_operator_metrics.md References • using_starrocks/caching/block_cache.md • data_source/data_cache.md • administration/management/monitoring/metrics-shared-data.md

@Rocky I use version 3.5.16 how can I catch query profile your version is not correct

In StarRocks v3.5.16, the command

SHOW PROFILE FOR QUERY "id"

is indeed deprecated or may not return the detailed information you need. The modern and recommended ways to capture the profile in the 3.x series are as follows: 1. Using

get_query_profile

(Best for raw text) This is the most reliable way to get the full execution profile via SQL.

sql
-- 1. Enable profiling for the session
SET enable_profile = true;

-- 2. Execute your query
SELECT * FROM your_iceberg_table LIMIT 10;

-- 3. Get the Query ID of the last query
SELECT last_query_id(); 
-- Output example: 019b364f-10c4-704c-b79a-af2cc3a77b89

-- 4. Retrieve the profile (use \G if using the mysql command line)
SELECT get_query_profile('019b364f-10c4-704c-b79a-af2cc3a77b89')\G

2. Using

ANALYZE PROFILE

(Best for readability) If you want a visualized, text-based summary that highlights bottlenecks (like S3 latency), use this command. It breaks down time spent per operator.

sql
-- Use the Query ID obtained from last_query_id() or SHOW PROFILELIST;
ANALYZE PROFILE FROM 'your_query_id';

3. Using

EXPLAIN ANALYZE

(One-step) You can skip the query ID entirely by prefixing your query. This will execute the query and immediately return the analyzed profile.

sql
EXPLAIN ANALYZE SELECT * FROM your_iceberg_table LIMIT 10;

4. Via the FE Web UI (Easiest for navigation) 1. Go to

http://<FE_IP>:<FE_HTTP_PORT>

(default port is

8030

). 2. Click on the Queries tab at the top. 3. Find your query in the list and click the link in the Profile column. Why you should look at the "CONNECTOR_SCAN" node: Since you are investigating S3 latency for external catalogs, search for

CONNECTOR_SCAN_NODE

in the profile output. Specifically, look for: *

DataCacheWriteTimer

: The actual time spent waiting for S3 to respond and writing to local cache. *

ScanTime

: The total time the scan operator was active. *

WaitPushDownV2Timer

: If this is high, the system is waiting for Iceberg metadata/manifest processing. If these metrics show high values while

fslib

metrics remain zero, it confirms the I/O is bypassing the Starlet internal volume layer (standard for external catalogs) and you should rely on the Data Cache/Block Cache metrics instead. References • best_practices/query_tuning/query_profile_overview.md • best_practices/query_tuning/query_profile_text_based_analysis.md • best_practices/query_tuning/query_planning.md

@Rocky so scan time is the real latency metric ?

No,

Scan Time

is not the precise S3 latency metric—it is a "top-level" bucket that includes several different operations. If you want to isolate S3 network performance, you need to look deeper into the Scan Breakdown. Based on your profile, here is the breakdown of what is actually happening: 1.

FS IO Time

= The "Real" S3 Latency In your profile, under the

Input Stream

section: *

FS IO Time

(158.661 ms): This is the actual time the StarRocks BE spent waiting for the S3 API to return data. This is the metric you should track if you suspect S3 itself is slow. 2.

Column Read Time

= CPU Processing *

Column Read Time

(228.555 ms): This is the time spent opening the Parquet files, reading the footers, and decoding the actual columns from the S3 data. * Note that this is often higher than

FS IO Time

because StarRocks is doing CPU-intensive work (unzipping/decoding) while the data is streaming in. 3.

Scan Time

= The Operator Wall-Clock Time *

Scan Time

(402.006 ms): This is the cumulative time the scan threads were active. It is effectively:

FS IO Time

+

Column Read Time

+

Buffer Management Overhead

. * In your case,

Scan Time

is roughly double your S3 latency because the BE spent a significant amount of time processing (decoding) the data it received from S3. 4. Why

fslib

is still zero As mentioned previously, the

fslib_read_io_latency

metric in Prometheus is hard-wired to the Starlet library used for StarRocks' own internal storage (Cloud-native tables). For Iceberg/External Catalogs, StarRocks uses a different I/O path. To monitor this via Prometheus instead of manual profiles, you should look for: *

starrocks_be_connector_scan_node_total_timer

: Monitors the total time of external scans. *

starrocks_be_files_scan_num_bytes_read

: Monitors the throughput of external scans. Summary for your Investigation: If you are trying to prove S3 is the bottleneck: 1. Look at

FS IO Time

in the profile. 2. If

FS IO Time

is low (e.g., < 200ms) but

Scan Time

is high (e.g., > 1s), the bottleneck is likely your BE CPU (decoding Parquet) or Disk I/O (if using cache), not S3 network latency to S3. 3. In your specific profile, S3 (158ms) and Parquet Decoding (228ms) are fairly balanced, which is healthy behavior for a query without a cache. References • best_practices/query_tuning/query_profile_operator_metrics.md • administration/management/monitoring/metrics.md • administration/management/monitoring/metrics-shared-data.md References • best_practices/query_tuning/query_profile_operator_metrics.md • loading/loading_introduction/troubleshooting_loading.md

@Rocky where I can find this metrics in your template overview dashboard?

In the standard StarRocks Grafana Dashboard (the "All-Architecture" or "BE" templates), you won't find these in the top-level "Overview" summary. You need to expand specific sections in the BE Dashboard: 1. Where to find "Files Scan" (Throughput) Look for a row labeled "Connector" or "External Scan". * Panel Name: Usually "Connector Scan Throughput" or "External Scan Bytes". * Metric used:

starrocks_be_files_scan_num_bytes_read

* What it tells you: This is the equivalent of the "S3 Throughput" for your Iceberg/Delta queries. If this is high but your query is slow, S3 is delivering data, but your BE might be struggling to process it. 2. Where to find "Scan Latency" In the same "Connector" or "BE - Storage" row: * Panel Name: "Connector Scan Node Total Time". * Metric used:

starrocks_be_connector_scan_node_total_timer

* What it tells you: This is the cumulative time the BE spent on the scan operator. It correlates with the "Scan Time" you see in your query profile. 3. Why you can't find "FS IO Time" in the Dashboard The metric

FS IO Time

(the 158ms in your profile) is currently not exported as a standalone Prometheus metric in the default templates for external catalogs. * Because StarRocks uses a thread-pool-based asynchronous scanner for Iceberg/Delta, this fine-grained I/O wait time is captured in the Query Profile but is typically aggregated into the total scan time for Prometheus. * To see the "Network Latency" trend for S3 without looking at profiles, you usually have to look at your Cloud Provider's S3 metrics (e.g., AWS CloudWatch for S3

TotalRequestLatency

). Summary of Dashboard Locations: Information | Dashboard Section | Panel Name S3 Data Volume |

BE

->

Connector

| Connector Scan Bytes / Throughput Scan Performance |

BE

->

Connector

| Connector Scan Node Total Time Cache Hits |

BE

->

Data Cache

| Data Cache Read Bytes (Hit vs Miss) Pro Tip: If your dashboard is missing the "Connector" row, ensure you are using the latest JSON template from the StarRocks GitHub or documentation, as these metrics were added in later 3.x versions. References • administration/management/monitoring/metrics.md