@Rocky How do I “*Check Disk I/O:* Since you are in a shared-data architecture, check the I/O and CPU of your CN nodes. High “Wait IO” often slows down the PK index updates.” using Grafana?

In a StarRocks shared-data architecture, Primary Key (PK) index updates occur during the Commit stage of a load. If you are seeing slow ingestion, checking the CPU and Disk I/O of your CN (Compute Nodes) is critical because high "Wait IO" often indicates that the local disk (used for PK index persistence and data caching) is becoming a bottleneck. To check these in Grafana, use the following metrics and dashboard sections: 1. Check "Wait IO" (CPU IOWait) Wait IO indicates the percentage of time the CPU is idle while waiting for an outstanding disk I/O request. * Metric Name:

starrocks_be_cpu_iowait

(CN nodes report under the

be

prefix in Prometheus). * Grafana Panel: Look for the "CPU" or "System Load" row in the standard StarRocks BE or Shared-data dashboard. * What to look for: If this value is consistently high (e.g., >10-20%), it means the CN is waiting on the local disk. Since PK tables in shared-data clusters (v3.2+) often persist their index to local storage to save memory, disk latency directly slows down the index lookups and updates required for each write. 2. Check Disk I/O Utilization * Metric Name:

starrocks_be_disk_io_util

or

starrocks_be_disk_io_time_ms

. * Grafana Panel: Look for the "Disk" row. The "Disk Utility" panel shows the percentage of time the disk was busy. * What to look for: Utilization near 100% indicates the disk is saturated. This is common if the

storage_root_path

is on a slow HDD or if the local SSD is overwhelmed by simultaneous data caching and PK index updates. 3. Monitoring Primary Key Index Specifically If you suspect the PK index is the issue, you can monitor its memory and I/O impact: * Memory Tracker: Check the "Update" item in the "Memory" panel of the dashboard (or via

http://<cn_ip>:<cn_http_port>/mem_tracker?type=update

). This tracks the memory used by the PK index. * Shared-data Dashboards: StarRocks provides specialized dashboards for shared-data: * Shared-data Dashboard: Focuses on transaction and publish version latency. * Starlet Dashboard: Focuses on

fslib

(S3 and Cache) I/O metrics. Look for "FSLIB READ/WRITE IO LATENCY" to see if local cache operations are slow. Summary of Actions If you find high Wait IO: 1. Verify Disk Speed: Ensure the

storage_root_path

on your CN nodes is on high-performance SSDs (NVMe preferred). 2. Adjust Concurrency: If the disk is saturated, you may need to reduce the

transaction_publish_version_worker_count

in

cn.conf

(default is the number of CPU cores) to reduce parallel I/O pressure during the commit stage. 3. Check PK Index Persistence: If memory is not an issue but I/O is, ensure you haven't set

l0_max_mem_usage

too low, as this forces more frequent index persistence to disk. References • administration/management/monitoring/metrics-shared-data.md • administration/management/monitoring/metrics.md • administration/management/monitoring/Monitor_and_Alert.md • using_starrocks/caching/block_cache.md