@Rocky I am trying to figure out how much data I am transferring from my external cloud storage to StarRocks. I have the audit loader plugin installed on this cluster. Is there a good way to get that metric per day?

If you have the Audit Loader plugin installed, you can query its destination table (typically

starrocks_audit_db*.starrocks_audit_tbl*

) to find the volume of data scanned from external sources. 1. Using Audit Logs (SQL-based loads) If you are using

INSERT INTO ... SELECT FROM FILES()

or querying external tables (e.g., Hive, Iceberg, or S3 catalogs), the

scanBytes

field in the audit log captures the amount of data read. You can run this query to get the total daily transfer in GB:

sql
SELECT 
    DATE(timestamp) AS day,
    round(SUM(scanBytes) / 1024 / 1024 / 1024, 2) AS data_transferred_gb
FROM starrocks_audit_db__.starrocks_audit_tbl__
WHERE (stmt LIKE '%FILES(%' OR catalog != 'default_catalog')
  AND isQuery = 0 -- Focus on load/insert statements
GROUP BY day
ORDER BY day DESC;

2. Using Information Schema (Broker Loads) If you use Broker Load to transfer data from cloud storage, the metrics are stored in the

information_schema.loads

view. Note that

ScanBytes

is often nested within the

JOB_DETAILS

JSON column.

sql
SELECT 
    DATE(CREATE_TIME) AS day,
    round(SUM(CAST(get_json_object(JOB_DETAILS, '$.ScanBytes') AS BIGINT)) / 1024 / 1024 / 1024, 2) AS data_transferred_gb
FROM information_schema.loads
WHERE TYPE = 'BROKER' AND STATE = 'FINISHED'
GROUP BY day
ORDER BY day DESC;

3. Using System Metrics (Cluster-wide) For a more comprehensive view that includes all ingestion types (Stream Load, Routine Load, etc.), StarRocks exposes metrics that you can view via the FE/BE Prometheus endpoints or the API: * Metric Name:

starrocks_be_files_scan_num_bytes_read

(Total bytes read from external storage like S3/GCS/OSS). * API call: You can get a quick summary of data volume by running:

bash
    curl http://<FE_IP>:<FE_HTTP_PORT>/api/show_data

Recommendation * If you primarily use

FILES()

or External Catalogs, use the Audit Loader query. * If you primarily use Broker Load, use the

information_schema.loads

query. * For Stream Load, the Audit Loader doesn't typically capture the byte count of the HTTP body; you should use the BE metric

streaming_load_bytes

. References • administration/management/monitoring/metrics.md • administration/management/logs.md • administration/management/audit_loader.md