@Rocky where to check brpc_idle_timeout_sec this?

In StarRocks,

brpc_idle_timeout_sec

is a configuration parameter for the Backend (BE) (and CN) nodes. It was introduced to manage how long idle bRPC connections remain open, primarily to help resolve issues with excessive "connection timed out" logs in certain network environments (like Kubernetes). You can check and manage this parameter in the following locations: 1. Through the BE HTTP API (Recommended) You can view the current configuration of a running BE node by accessing its configuration endpoint:

bash
curl http://<BE_IP>:<BE_HTTP_PORT>/be_config

* The default

BE_HTTP_PORT

is 8040. * Look for

brpc_idle_timeout_sec

in the output. 2. In the

be.conf

File If you want to set or check the persistent value, look in the configuration file located at

be/conf/be.conf

on your BE/CN nodes. * If it is not present, it is using the default value (often

-1

, which means no timeout, though this can vary by version). * To change it, add or modify the line:

brpc_idle_timeout_sec = <value>

and restart the BE. 3. Through the bRPC Internal Status Page Since this is a bRPC-specific flag, you can also view it on the bRPC internal web interface:

text
http://<BE_IP>:<BRPC_PORT>/flags

* The default

BRPC_PORT

is 8060. * Search for

idle_timeout_second

or similar on this page. Important Note This parameter is relatively new (introduced/enhanced around StarRocks v3.1+ in recent patches or v3.3/v3.4 series). If you do not see it in your configuration, your version might be older, or you may need to use the bRPC internal

/flags

endpoint to see if the underlying bRPC library supports it. Common use case: Setting this to a value like

10

or

30

can help reduce

socket.cpp:1379] Fail to wait EPOLLOUT

error logs in environments with aggressive network idle timeouts. References • administration/management/FE_configuration.md • administration/management/BE_configuration.md

@Rocky in 8060/flags I have

idle_timeout_second = 30 secs

but I get errors like that in cn W20260810 210826.226856 134964106212928 socket.cpp:1379] Fail to wait EPOLLOUT of fd=908: Connection timed out [110]

The presence of

idle_timeout_second = 30 secs

in your bRPC flags confirms the parameter is active. However, if you are still seeing

Fail to wait EPOLLOUT ... Connection timed out [110]

, it usually means the network infrastructure (like a Kubernetes LoadBalancer, Istio sidecar, or cloud firewall) is closing idle connections sooner than 30 seconds, or the connection is being dropped silently. To resolve this, try the following steps: 1. Reduce the timeout further If your network environment has an aggressive idle timeout (e.g., some cloud providers or ingress controllers default to 30s or even 15s), your StarRocks setting must be lower than the network's limit. * Try setting

brpc_idle_timeout_sec = 10

or even

brpc_idle_timeout_sec = 5

in your

cn.conf

(and

be.conf

). * This ensures StarRocks proactively closes the connection before the network layer cuts it off, which prevents the "Connection timed out" error when StarRocks later tries to reuse that socket. 2. Check the FE side The error

Fail to wait EPOLLOUT

on a CN often happens when the CN is trying to send data back to the FE or another BE. * Check if your FE nodes also have this parameter set. If the FE is the one initiating a request on a connection that the network has already killed, the sender will log the EPOLLOUT error. * In FE, the equivalent parameter is often managed via

brpc_idle_wait_max_time

(default 10000ms/10s), but verify if your version supports

brpc_idle_timeout_sec

in

fe.conf

. 3. Increase Socket Buffer Limits In some high-concurrency scenarios, the "timeout" is actually a side effect of the socket buffer being full (leading to "server overcrowded" or EPOLLOUT failures). * Add this to your

cn.conf

and `be.conf`:

properties
    brpc_socket_max_unwritten_bytes = 1073741824

(Setting this to 1GB or higher can help if the failure occurs during heavy data transmission). 4. Verify Network Environment If you are on Kubernetes: * Check your Service or Ingress annotations for

idle-timeout

. * If using a Service Mesh (like Istio), check the

destinationRule

or

meshConfig

idle timeouts. * If the "Connection timed out" occurs exactly after a period of inactivity, the

brpc_idle_timeout_sec

must be set to a value strictly smaller than that period. Summary Recommendation: Decrease

brpc_idle_timeout_sec

to 10 in

cn.conf

and restart the CN. If the logs persist, apply the same change to your FE and all BE nodes. References • administration/management/FE_configuration.md • knowledge/usage/best-practice.md • faq/loading/Loading_faq.md • administration/management/BE_configuration.md • faq/Sql_faq.md

@Rocky I have a problem that when writing data via spark starrocks jar numrunning queries increases indefinetely and eats all the ram eventhough they finish writes

This behavior (where

num_running_queries

grows indefinitely and causes OOM even though Spark reports writes are "finished") is a known symptom of connection/transaction leakage in StarRocks, often exacerbated by the network timeout issues you mentioned. When you see

Fail to wait EPOLLOUT

, it means the network is dropping connections before StarRocks or Spark can gracefully close them. Because the connection is "zombied," the StarRocks FE/BE may keep the associated "Query" (which, for a write, is the Stream Load task) in a

RUNNING

state waiting for more data or a commit signal that never arrives. Here is how to troubleshoot and fix this: 1. Fix the bRPC "Zombie" Connections (Immediate Action) As discussed, the

EPOLLOUT

error proves your network (K8s, LB, or Firewall) is killing idle sockets. If StarRocks doesn't proactively close these, the internal "Query" object stays in memory. * Set

brpc_idle_timeout_sec = 10

in

be.conf

(and

cn.conf

). * Set

brpc_idle_wait_max_time = 10000

in

fe.conf

. * Restart the nodes. This forces StarRocks to drop idle connections before the network kills them, ensuring the "Running Query" context is cleaned up. 2. Check for Leaked Transactions If you are writing via the Spark connector, it likely uses the Stream Load Transaction Interface (v2.5+). If a write "finishes" in Spark but a network glitch prevents the

COMMIT

call from reaching StarRocks, the transaction stays open. * Run this in StarRocks:

sql
    SHOW PROC '/transactions';
    -- or for a specific database:
    SHOW TRANSACTIONS FROM your_db_name;

* Look for transactions in the

PREPARE

state that are very old. These transactions count toward resource limits and consume memory. * If you see many of these, you can manually abort them or wait for the

stream_load_default_timeout_second

(default 600s) to expire. 3. Adjust Spark Connector Parameters The Spark connector (version 1.1.1+) changed its default behavior to mitigate these issues. * Check your version: Ensure you are using

starrocks-spark-connector

1.1.1 or later. * Disable Transaction Load (Optional): If you don't need exactly-once 2PC, using standard Stream Load is more resilient to network drops. * Set

starrocks.write.max.retries

to a positive value (e.g.,

3

). * In 1.1.1+, this automatically switches the connector to the standard Stream Load interface, which is less likely to leave "hanging" queries. * Reduce Buffer Rows: If Spark is buffering too much data before sending, it can lead to long-running single loads. Try:

properties
    starrocks.write.buffer.rows = 100000

4. Verify "Running Queries" Source To see exactly what these "phantom" queries are, run:

sql
SHOW PROC '/current_queries';

Look at the

StartTime

and

User

. If the

StartTime

is from hours ago, they are definitely leaked contexts. Check the

QueryId

and match them against the FE logs (

fe.log

) to see if they end with an error like

Label already exists

or

Transaction not found

. Summary Recommendation The OOM is caused by the FE/BE holding onto metadata and buffers for "Running" loads that are actually dead. 1. Lower

brpc_idle_timeout_sec

to 10. 2. Verify that your Spark jobs are not creating an excessive number of partitions (each partition = one stream load = one running query). Try to

coalesce

or

repartition

your Spark DataFrame to a reasonable number before writing. References • unloading/Spark_connector.md • using_starrocks/running_queries.md

@Rocky [42000][1064] Getting syntax error at line 1, column 45. Detail message: No viable statement for input 'SHOW TRANSACTIONS'.

The error

[42000][1064] ... No viable statement for input 'SHOW TRANSACTIONS'

occurs because

SHOW TRANSACTIONS

(plural) is not a valid top-level command in StarRocks. To view transaction information, you must use the

SHOW PROC

interface or the singular

SHOW TRANSACTION

command for a specific ID. 1. How to list transactions To see the status of transactions across your databases, use the following steps: * Step 1: List databases with transactions

sql
    SHOW PROC '/transactions';

This returns a list of

DbId

and

DbName

. * Step 2: Check transaction counts for a specific DB Replace

<db_id>

with the ID from Step 1:

sql
    SHOW PROC '/transactions/<db_id>';

This shows the count of

running

and

finished

transactions in that database. * Step 3: List detailed running transactions

sql
    SHOW PROC '/transactions/<db_id>/running';

This will list the actual

TransactionId

,

Label

, and

PrepareTime

of the active loads. 2. How to view a specific transaction If you have a

TransactionId

(e.g., from the previous step or an error log), use the singular command:

sql
SHOW TRANSACTION FROM <db_name> WHERE id = <transaction_id>;

3. Check "Running Queries" (The OOM culprit) Since you mentioned

num_running_queries

is growing, these might be Stream Load sessions that haven't timed out yet. Check them here:

sql
SHOW PROC '/current_queries';

If you see many long-running queries from the user associated with Spark, look at the

StartTime

. If they have been running for hours, they are likely "leaked" because the network dropped the connection before the Spark connector could send the final commit signal. Summary Checklist 1. Reduce Timeout: As mentioned, set

brpc_idle_timeout_sec = 10

in `be.conf`/`cn.conf`. 2. Clean up: If you find thousands of running transactions in

SHOW PROC '/transactions/<db_id>/running'

, they will eventually time out based on your

stream_load_default_timeout_second

, but you may need to restart the FE if the memory pressure is too high. 3. Spark Config: Ensure your Spark connector is version 1.1.1+ or newer and check that it isn't creating too many small partitions (which creates too many concurrent transactions). References • sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.md • sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.md • loading/SQL_transaction.md • sql-reference/sql-statements/loading_unloading/SHOW_LOAD.md • sql-reference/sql-statements/materialized_view/SHOW_MATERIALIZED_VIEW.md • sql-reference/sql-statements/loading_unloading/unloading/SHOW_EXPORT.md • sql-reference/sql-statements/account-management/SHOW_GRANTS.md