@Rocky whats that starrocks_fe_txn_reject

starrocks_fe_txn_reject

is a monitoring metric in StarRocks that tracks the total number of transactions rejected by the Frontend (FE). When this metric increments, it typically means your cluster has reached a safety limit for concurrent transactions, and the FE is refusing new requests to prevent system instability. Common Cause:

max_running_txn_num_per_db

The most frequent reason for transaction rejections is reaching the limit of concurrent load transactions allowed for a single database. * The Error: You will often see a corresponding error message in your logs or client:

"current running txns on db XXX is 1000, larger than limit 1000"

. * The Limit: Controlled by the FE configuration parameter

max_running_txn_num_per_db

. * In older versions (pre-v3.1), the default was 100. * In newer versions (v3.1+), the default was increased to 1000. How to Fix It 1. Check the Current Limit: Run

ADMIN SHOW FRONTEND CONFIG LIKE 'max_running_txn_num_per_db';

to see your current setting. 2. Increase the Limit (If Necessary): If you have a high-concurrency ingestion requirement, you can increase this limit dynamically:

sql
    ADMIN SET FRONTEND CONFIG ("max_running_txn_num_per_db" = "2000");

Note: Increasing this too high can lead to increased memory consumption and lock contention on the FE. 3. Optimize Load Frequency: If you are hitting this limit due to many small "Stream Loads," consider batching your data into fewer, larger load requests to reduce the transaction count. Where to find it You can view this metric by accessing the FE Prometheus endpoint (usually

http://<fe_ip>:<http_port>/metrics

) and searching for

starrocks_fe_txn_reject

. References • faq/loading/Loading_faq.md • administration/management/FE_configuration.md • administration/management/BE_configuration.md • loading/SQL_transaction.md

@Rocky what about this one starrocks_be_stream_load

starrocks_be_stream_load

is a cumulative counter metric on the Backend (BE) that tracks the total volume of data processed via Stream Load. Unlike metrics that track request counts, this metric typically provides two specific data points via labels: *

starrocks_be_stream_load{type="load_rows"}

: The total number of rows successfully loaded. *

starrocks_be_stream_load{type="receive_bytes"}

: The total number of bytes received by the BE for Stream Load jobs. Key Related Metrics While

starrocks_be_stream_load

tracks the volume of data, you often use it alongside these other BE metrics to monitor ingestion health: *

starrocks_be_streaming_load_requests_total

: The total number of Stream Load requests (successes + failures). *

starrocks_be_streaming_load_duration_ms

: The cumulative time spent processing Stream Loads, used to calculate average latency. *

starrocks_be_streaming_load_current_processing

: A gauge showing how many Stream Load jobs are running right now on that BE. *

starrocks_be_stream_load_pipe_count

: The number of active data pipes/streams currently open. Usage Tip If you see

receive_bytes

increasing but

load_rows

staying flat, it may indicate that the BE is receiving data but failing to parse or commit it (check

starrocks_be_streaming_load_requests_total

with error labels if available, or the

ErrorURL

in the Stream Load response). References • administration/management/monitoring/metrics.md • loading/StreamLoad.md • administration/management/BE_configuration.md References • loading/StreamLoad.md • sql-reference/sql-statements/loading_unloading/STREAM_LOAD.md • administration/management/BE_configuration.md

@Rocky is that connected to txn?

Yes,

starrocks_be_stream_load

is directly connected to transactions. In StarRocks, every Stream Load is treated as a single transaction. Here is how they are connected: 1. The Transaction Lifecycle When you perform a Stream Load, the following happens: * FE (Frontend): Opens a transaction (Txn) and assigns a

TxnID

. This is where the

starrocks_fe_txn_reject

metric applies—if the FE has too many open transactions, it will reject the load before it even starts. * BE (Backend): Receives the data and writes it to disk. The metric

starrocks_be_stream_load

tracks the rows and bytes flowing through the BE during this process. * FE (Frontend): Once the BE finishes, the FE commits the transaction, making the data visible. 2. Transactional Stream Load vs. Standard Stream Load There are actually two sets of metrics on the BE depending on how you load: *

starrocks_be_stream_load

: Tracks standard Stream Loads (where one HTTP request = one transaction). * *`starrocks_be_transaction_streaming_load_*`*: Tracks the newer Stream Load Transaction Interface, which allows you to send multiple HTTP requests (data chunks) under a single

TxnID

before committing. 3. Why it matters for troubleshooting If you see

starrocks_be_stream_load

(bytes) increasing but your data isn't appearing in the table: * The transaction likely failed at the Commit stage. * Check the FE logs for "transaction" errors or look at the

starrocks_fe_txn_reject

metric to see if the system is refusing to finalize the work the BE just did. Summary:

starrocks_be_stream_load

measures the work (data) being done, while "txn" metrics measure the management (start/stop/limits) of that work. References • administration/management/monitoring/metrics.md • loading/StreamLoad.md • loading/StreamLoadTransactionInterface.md • administration/management/FE_configuration.md References • administration/management/monitoring/metrics.md

@Rocky whats that starrocks_fe_request_total

starrocks_fe_request_total

is a cumulative monitoring metric that tracks the total number of requests received by the StarRocks Frontend (FE). It is closely related to, but distinct from,

starrocks_fe_query_total

. Key Differences *

starrocks_fe_request_total

(RPS - Requests Per Second): This counts all protocol-level requests sent to the FE. This includes not just SQL queries, but also connection handshakes, metadata operations,

SET

statements, and other administrative commands that might not trigger a full query execution plan. *

starrocks_fe_query_total

(QPS - Queries Per Second): This counts specific SQL queries (like

SELECT

,

INSERT

,

UPDATE

,

DELETE

). It represents the actual analytical or data-manipulation workload. Why Monitor Both? Monitoring both helps you distinguish between connection/protocol overhead and actual query workload: 1. High Request Count vs. Low Query Count: If

request_total

is much higher than

query_total

, it often indicates a "chatty" application that is opening/closing connections frequently or executing many small metadata commands (like

SHOW TABLES

or

SET

session variables) rather than running actual data queries. 2. Resource Bottlenecks: A spike in

request_total

without a spike in

query_total

might point to a problem with the connection pooler or a health check script that is hitting the FE too aggressively. Common Labels This metric often includes labels to help you break down the requests: *

status

: May distinguish between successful and failed requests. *

type

: (In some versions/exporters) May categorize the request type. How to use it in Grafana To calculate the Requests Per Second (RPS) for your cluster, you would typically use a PromQL query like:

promql
sum(irate(starrocks_fe_request_total[1m]))

References • administration/management/monitoring/metrics.md • administration/management/FE_configuration.md • administration/management/monitoring/Monitor_and_Alert.md References • administration/management/monitoring/metrics.md