@Rocky I have a flink stream load that writes into a sink table in SR shared data cluster (4.1.3). But when using this prometheus metric we are

starrocks_fe_table_load_rows

we are not observing any data points for these tables. Is there another equivalent metric we can use for getting table rows increase?

The metric

starrocks_fe_table_load_rows

is indeed the correct one for tracking row increases at the table level from the Frontend (FE). However, there are a few reasons why you might not be seeing data points for it in your cluster (StarRocks 4.1.3—though note that as of late 2024, the current stable version is v3.3.x, so you may be using a specific distribution or meant v3.1.3). 1. Known Limitation with Flink (Transaction Interface) A known issue (tracked as #44991) existed where the Stream Load transaction interface (which the StarRocks Flink Connector uses for exactly-once loading) did not populate the

starrocks_fe_table_load_rows

and

starrocks_fe_table_load_bytes

metrics. * Status: This was officially fixed in StarRocks v3.3. * Action: If you are actually on a version prior to 3.3 (like v3.1.3 or v3.2.3), these metrics will remain empty for Flink stream loads. You would need to upgrade to v3.3+ to use this specific metric with Flink. 2. FE Configuration Requirement Even in supported versions, table-level metrics must be explicitly enabled in the FE configuration. * Parameter:

enable_table_metrics_collect

* Default:

true

(but check your

fe.conf

or run

ADMIN SHOW FRONTEND CONFIG;

) * If this is set to

false

, the FE skips exporting table-level metrics to the

/metrics

endpoint to reduce overhead. 3. Equivalent/Alternative Metrics If the FE table metric is not available for your current version, you can use these alternatives: * BE/CN Node Metrics (Cluster Level): On your Compute Nodes (CN), look for the following metrics. These provide an aggregate view of loading activity: * `starrocks_be_load_rows`: Total rows loaded by all methods (Stream Load, Broker Load, etc.) on that node. * `starrocks_be_stream_load_bytes`: Total bytes received via Stream Load. * `starrocks_be_stream_load_requests_total`: Total number of Stream Load requests. * Flink Connector Metrics: The StarRocks Flink Connector itself exports metrics to the Flink monitor. You can track: * `totalRows`: Total rows sent to StarRocks. * `totalBytes`: Total bytes sent. These are often more reliable for monitoring the health of the stream itself. * System Tables (SQL-based): If you need to verify row increases via SQL (for a dashboard or script), you can query the

information_schema.stream_loads

table:

sql
    SELECT TABLE_NAME, SUM(NUM_ROWS_NORMAL)
    FROM information_schema.stream_loads
    WHERE CREATE_TIME_MS > ...
    GROUP BY TABLE_NAME;

Summary Recommendation * Check your version: if you are on a version below v3.3,

starrocks_fe_table_load_rows

will not work with Flink. * Use CN metrics: Monitor

starrocks_be_load_rows

on your Compute Nodes for a cluster-wide view of ingestion. * Verify Config: Ensure

enable_table_metrics_collect = true

is set in your FE configuration. References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md

@Rocky my starrocks version is 4.1.3

In StarRocks 4.1.3 (and other recent versions), the metric

starrocks_fe_table_load_rows

is the primary way to monitor table-level ingestion from the Frontend (FE). However, there is a known behavior when using the Flink StarRocks Connector (which utilizes the Stream Load Transaction Interface for exactly-once semantics). Why the metric is empty Even in modern versions like 4.1.3, the Transaction Interface (used by Flink to

BEGIN

,

SEND

, and

COMMIT

transactions) sometimes does not correctly propagate row counts to the FE's table-level Prometheus metrics collector. This is a known limitation (historically tracked as #44991) where the FE metric excludes data loaded via this specific transaction-based stream load path. Equivalent/Alternative Metrics Since the FE table-level metric is not populating for your Flink stream, you can use the following alternatives to monitor row increases: 1. CN Node Metrics (Cluster Level) Instead of the FE, look at the metrics exported by your Compute Nodes (CN). These provide the actual work performed by the nodes, though they are usually aggregated at the node level rather than the table level. *

starrocks_be_load_rows

: The total number of rows loaded by the node. In a shared-data cluster, this will be reported by the CN nodes. *

starrocks_be_stream_load_bytes

: The total bytes received via the stream load interface. *

starrocks_be_load_rpc_count

: Total load requests. 2. SQL-based Monitoring (Table Level) If you need table-specific row counts for a Grafana dashboard or report, the most reliable source in StarRocks is the

information_schema

. You can query this periodically:

sql
-- Get total rows loaded per table in the last hour
SELECT 
    TABLE_NAME, 
    SUM(NUM_ROWS_NORMAL) AS rows_loaded
FROM information_schema.stream_loads
WHERE CREATE_TIME > NOW() - INTERVAL 1 HOUR
  AND STATUS = 'VISIBLE'
GROUP BY TABLE_NAME;

Note: If you use a Prometheus SQL Exporter, you can turn this query into a custom Prometheus metric. 3. Flink Connector Metrics (Source of Truth) The StarRocks Flink Connector itself exposes metrics to the Flink monitoring system. If you have Prometheus monitoring set up for your Flink cluster, you should use these: *

totalRows

: The total number of rows sent to StarRocks. *

totalBytes

: The total volume of data sent. These are often more accurate for streaming jobs as they represent the data being pushed into the sink in real-time. Configuration Check If you still want to try enabling the FE metrics (in case it's a configuration issue), ensure the following parameter is set to

true

in your

fe.conf

or via `ADMIN SHOW FRONTEND CONFIG`: *

enable_table_metrics_collect

: Must be

true

(default is

true

). *

txn_latency_metric_report_groups

: Ensure

stream_load

is included in this list if you are looking for transaction-related latency/counts. Summary Recommendation For a Shared-data cluster (4.1.3) using Flink, the FE table-level metric is likely staying at 0 due to the transaction interface. * Use Flink's internal metrics for table-level row counts. * Use

starrocks_be_load_rows

on your CN nodes to monitor the overall cluster ingestion health. * Use a query against

information_schema.stream_loads

if you require a StarRocks-side view of table-specific ingestion history. References • administration/management/monitoring/metrics.md • administration/management/monitoring/alert.md