@Rocky is there any variables i can get from query profile for checking s3 get calls?

In StarRocks, you can monitor S3 GET calls (and other remote storage interactions) using specific metrics in the Query Profile. The variables available depend on whether you are using a shared-data cluster or external tables with Data Cache. Here are the primary variables to look for in the

OLAP_SCAN

or

CONNECTOR_SCAN

sections of the profile: 1. Shared-data Cluster Metrics If you are running StarRocks in shared-data mode (where S3 is the primary storage), the following variables directly track remote S3 operations: *

IOCountRemote

: The number of I/O operations (S3 GET calls) made to remote storage. *

CompressedBytesReadRemote

: The total amount of compressed data read from S3. *

IOTimeRemote

: The total time spent waiting for remote I/O operations to complete. *

SegmentsReadCount

: While not a direct S3 count, a high number here indicates many remote files are being opened. 2. Data Cache Metrics When Data Cache is enabled (common for both shared-data and external tables like Iceberg/Hive on S3), the system records cache misses as "writes" from remote storage to local disk/memory. These are excellent proxies for S3 GET calls: *

DataCacheWriteCounter

: The number of times data was fetched from S3 and written to the local cache. This is essentially your S3 GET call count. *

DataCacheWriteBytes

: The total bytes fetched from S3 to be cached locally. *

DataCacheWriteTimer

: The time spent downloading and writing that data from S3. *

DataCacheReadBytes

: Data read from the local cache. If

BytesRead

is much larger than

DataCacheReadBytes

, the difference was fetched from S3. 3. General Scan Metrics For external tables (Connector Scan) without specific remote counters, look at these standard metrics: *

ScanTime

: The total time taken for the scan operation. *

IOTaskExecTime

: The execution time of I/O tasks. If this is high and your hit rate is low, it usually points to S3 latency. *

BytesRead

: The total amount of data retrieved by the operator. Summary Table for Quick Reference Metric | What it represents

IOCountRemote

| Number of S3 GET requests (Shared-data mode).

DataCacheWriteCounter

| Number of S3 GET requests caused by cache misses.

CompressedBytesReadRemote

| Volume of data pulled from S3 (Shared-data mode).

DataCacheWriteBytes

| Volume of data pulled from S3 due to cache misses.

IOTimeRemote

| Latency of S3 GET calls. Rocky's Tip: To see these metrics, ensure you have set

SET enable_profile = true;

before running your query. You can then view the profile using

get_query_profile(query_id)

. References • best_practices/query_tuning/query_profile_operator_metrics.md • data_source/data_cache.md • administration/management/monitoring/metrics.md • using_starrocks/Cost_based_optimizer.md