@Rocky Suggest 2-3 more metrics (on top of rows per second ingested and load throughput (bytes)) for my grafana ingestion health dashboard overview. I am using the stat visualization. Shared nothing cluster with ingestion via stream load

For a Stream Load health dashboard on a shared-nothing cluster, monitoring just rows and throughput tells you "what" is happening, but not "how well" the system is handling it. On top of your current metrics, I suggest adding these 2-3 metrics to your stat visualizations: 1. Average Load Latency Metric:

rate(starrocks_be_streaming_load_duration_ms[1m]) / rate(starrocks_be_streaming_load_requests_total[1m])

* Why it matters: This tells you the average time (in ms) it takes to complete a single Stream Load. * Health Indicator: If throughput is steady but latency spikes, it usually indicates disk I/O bottlenecks or compaction backpressure on the BEs. It’s an early warning sign before loads start failing. 2. Failed Transaction Rate Metric:

rate(starrocks_fe_txn_failed[5m])

* Why it matters: Stream Load is transactional. This metric tracks transactions that were aborted due to timeouts, label conflicts, or internal errors. * Health Indicator: In a healthy system, this should be 0. A sudden rise often indicates network issues between your client and the FE, or that your

streaming_load_rpc_timeout_ms

is set too low for the data volume. 3. Current Concurrent Loads Metric:

starrocks_be_streaming_load_current_processing

* Why it matters: StarRocks has limits on how many concurrent loads can run (controlled by

max_running_txn_num_per_db

). * Health Indicator: Use this in a Gauge or Stat component. If this number nears your configured limit (default is often 100 or 1000 depending on version), new Stream Load requests will be rejected with a "current running txns reach limit" error. Bonus: Tablet Max Compaction Score Metric:

max(starrocks_be_tablet_cumulative_compaction_score)

* Why it matters: Ingestion creates new "versions" (rowsets). The BE must compact these into larger files. If the compaction score is high (e.g., > 100), it means the BE cannot keep up with the write rate. * Health Indicator: High scores will eventually trigger write throttling, causing your ingestion throughput to drop and latency to spike. Summary Table for Grafana: Metric Description | Prometheus Query (Example) Avg Latency |

rate(starrocks_be_streaming_load_duration_ms[1m]) / rate(starrocks_be_streaming_load_requests_total[1m])

Failures |

sum(rate(starrocks_fe_txn_failed[1m]))

Active Loads |

sum(starrocks_be_streaming_load_current_processing)

References • administration/management/monitoring/alert.md • loading/Stream_Load_transaction_interface.md • administration/management/monitoring/metrics.md • faq/loading/Loading_faq.md

@Rocky

setDefaultStorageVolumeStatement = `SET ` + storageVolumeName + ` AS DEFAULT STORAGE VOLUME;`

Technically is this needed? Setting default storage volume for the session? My application creates the storage volume, creates table linked to storage volume, then stream loads data to storage volume

Technically, the answer depends on whether you are referring to the cluster’s operation or a specific table creation. Here is the breakdown of why you might (or might not) need that statement: 1. It is Global, not Session-based First, a correction on scope:

SET <name> AS DEFAULT STORAGE VOLUME

is a global cluster-level setting, not a session-level setting. Once executed, it persists in the cluster metadata. You do not need to run it every time your application connects or for every load. 2. Is it needed for your specific Table/Load? No, it is not strictly needed for your specific table if you follow the workflow you described: * Explicit Table Property: If your

CREATE TABLE

statement includes

PROPERTIES ("storage_volume" = "your_volume_name")

, StarRocks will use that specific volume for that table, completely ignoring whatever the "default" volume is. * Stream Load: Stream Load targets a table. Once the table is created and linked to a volume, the load automatically goes to the storage associated with that table. 3. When is it "Technically" required? While not required for your specific table, a default storage volume is mandatory for the cluster's internal health if you are running in Shared-data mode: * System Statistics: StarRocks stores system metrics and statistics in the default storage volume. * Fallbacks: If a user creates a table without specifying the

storage_volume

property, it will fail unless a default is set. * Requirement: In a shared-data cluster, you must have at least one default storage volume configured for the cluster to function correctly. A Note on your Architecture You mentioned you are using a Shared-nothing cluster. There is a slight technical contradiction here: * Shared-nothing: Data is stored on the local disks of the BE nodes. You typically do not use

STORAGE VOLUME

or the

storage_volume

property for native tables in this mode. * Shared-data: Data is stored on object storage (S3/GCS/HDFS). This is where

STORAGE VOLUME

is the primary way to manage data locations. If your application is successfully using

PROPERTIES ("storage_volume" = "...")

, you are likely running a Shared-data (compute-storage decoupled) cluster, or you are using Storage Volumes for External Tables (like Iceberg or Hudi). Summary * If you specify the volume in `CREATE TABLE`: The

SET DEFAULT

statement is not needed for that table's functionality. * If you haven't set a default yet for the cluster: You must set one once (globally) so the cluster can store internal system data. * Recommendation: If your application logic ensures every table is explicitly linked to a volume, you only need to run the

SET DEFAULT

command once during your cluster's initial bootstrap/database's initial setup phase, not as part of your recurring application flow. References • sql-reference/sql-statements/cluster-management/storage_volume/SET_DEFAULT_STORAGE_VOLUME.md • sql-reference/sql-statements/table_bucket_part_index/CREATE_TABLE.md • deployment/shared_data/s3.md References • loading/StreamLoad.md • administration/management/monitoring/metrics.md • loading/Loading_intro.md • loading/Stream_Load_transaction_interface.md