<@U0A71G31CDV>, what does this exception mean? `fa...
# questions-and-troubleshooting
r
@Rocky, what does this exception mean?
fail to execute commit task: timeout by txn manager
What's a txn manager and is the timeout configurable?
r
This exception means that a transaction was aborted by the StarRocks Frontend (FE) Transaction Manager because it exceeded its allotted timeout period before completing. When you see
fail to execute commit task: timeout by txn manager
, it usually indicates that the data processing or the commit request took longer than the system was willing to wait. By the time the backend (BE) tried to finalize the transaction, the FE had already moved the transaction to an
ABORTED
state. What is the Transaction Manager? The Transaction Manager (specifically
GlobalTransactionMgr
in the FE) is the component responsible for managing the lifecycle of all data modification transactions (Stream Load, Routine Load, INSERT, etc.). It: * Assigns Transaction IDs. * Tracks the state of transactions (PREPARE, COMMITTED, VISIBLE, ABORTED). * Runs a background thread (
txnTimeoutChecker
) to automatically abort transactions that have been open for too long without finishing. Is the timeout configurable? Yes, the timeout is configurable, but the specific parameter depends on how you are loading data: 1. Routine Load (Kafka): * Configuration: It is set at the job level using the
TIMEOUT
property in your
CREATE ROUTINE LOAD
statement (default is usually 60s). * To fix: Alter the job to increase the timeout:
Copy code
sql
        ALTER ROUTINE LOAD FOR job_name PROPERTIES ("timeout" = "120");
2. Stream Load: * Configuration: Controlled by the
timeout
header in the HTTP request (default 600s). * Global Default:
stream_load_default_timeout_second
in
fe.conf
. 3. INSERT INTO SELECT: * Configuration: Controlled by the session variable
insert_load_default_timeout_second
(default 3600s). * To fix:
SET insert_load_default_timeout_second = 7200;
4. Global Constraints (fe.conf): Regardless of the job-specific settings, the FE enforces global limits in `fe.conf`: * `max_load_timeout_second`: The absolute maximum time any load transaction can run (default 3 days). * `min_load_timeout_second`: The minimum allowed timeout (default 1s). Troubleshooting Steps * Check Transaction Details: Run
SHOW TRANSACTION WHERE id = <txn_id>;
to see the exact
PrepareTime
and
TimeoutMs
for the failed transaction. * Identify Bottlenecks: If timeouts happen frequently despite high settings, check for: * High BE Load: Backends might be too busy to process the commit. * Lock Contention: Large deletes or schema changes can block transactions. * Network Issues: Slow RPC communication between FE and BE. Referencesadministration/management/BE_configuration.mdadministration/management/FE_configuration.md
r
@Rocky What's the correct syntax for this?
SET insert_load_default_timeout_second = 7200;
I get this when I try that:
Copy code
ERROR 1193 (HY000): Getting analyzing error. Detail message: Unknown system variable 'insert_load_default_timeout_second', the most similar variables are {'insert_timeout', 'cbo_enable_intersect_add_distinct', 'like_predicate_consolidate_min'}.
r
The error occurred because
insert_load_default_timeout_second
is an FE configuration parameter (set in the
fe.conf
file), not a session variable. For SQL statements, the correct session variable to use is
insert_timeout
. Correct Syntax To set the timeout for your current session:
Copy code
sql
SET insert_timeout = 7200;
To set it globally for all future sessions (requires ADMIN privileges):
Copy code
sql
SET GLOBAL insert_timeout = 7200;
Key Differences *
insert_timeout
(Session Variable):
This is what you should use in your SQL client. Since StarRocks v3.4.0, this variable specifically governs
INSERT
,
UPDATE
,
DELETE
, and
CTAS
operations, separating them from the standard
query_timeout
. *
insert_load_default_timeout_second
(FE Config):
This is a global default set in the
fe.conf
file on the Frontend nodes. It defines the initial default for the cluster, but is overridden by the session variable. Verification You can verify the current setting by running:
Copy code
sql
SHOW VARIABLES LIKE 'insert_timeout';
Referencesloading/InsertInto.mdsql-reference/System_variable.mdfaq/loading/Insert_into_faq.md Referencesloading/InsertInto.mdadministration/management/FE_configuration.mdloading/loading_introduction/loading_considerations.mdsql-reference/System_variable.mdsql-reference/sql-statements/loading_unloading/ETL/SUBMIT_TASK.mdfaq/loading/Insert_into_faq.mdsql-reference/sql-statements/loading_unloading/BROKER_LOAD.md
r
@Rocky, the transaction I was using which timed out was in a simple
INSERT
statement from our app over JDBC. The transaction was open for less than 15 minutes. I can see
insert_load_default_timeout_second
was set to its default value of
3600
, which should be an hour. Is there another config which could have caused the timeout?
r
The fact that your
insert_load_default_timeout_second
is set to 3600 (1 hour) but the task failed in ~15 minutes suggests that a session variable or a BE-side RPC timeout is overriding the global FE default. Here are the most likely configurations causing this: 1.
query_timeout
(The most common culprit)
In StarRocks versions earlier than v3.4.0, all
INSERT
statements were governed by the
query_timeout
session variable. * Default: 300 seconds (5 minutes). * Even if your global FE config is 1 hour, the session's
query_timeout
takes precedence for the statement execution. * Solution: Increase it for your session or globally:
Copy code
sql
    SET query_timeout = 3600;
2.
insert_timeout
(For v3.4.0+)
If you are on v3.4.0 or newer, StarRocks introduced
insert_timeout
specifically for DML operations to avoid conflicts with standard query timeouts. * If this is set to a lower value (e.g., 900 seconds = 15 minutes) in your session or application's JDBC connection string, it will override the FE default. * Solution:
SET insert_timeout = 3600;
3.
streaming_load_rpc_max_alive_time_sec
(BE Config)
Even if the FE Transaction Manager is willing to wait an hour, the Backends (BE) have their own internal RPC timeout for data loading tasks. * Default: 1200 seconds (20 minutes). * If the actual data writing phase on the BEs exceeds this limit, the BE may drop the task, leading the Transaction Manager to eventually report a timeout. * Check: Verify this in your
be.conf
. 4.
load_straggler_wait_second
(FE Config)
This parameter defines how long the FE waits for "straggling" replicas to finish their work during the commit phase. * Default: 300 seconds (5 minutes). * If your cluster is under high load and some nodes are much slower than others, the transaction might be aborted if it can't reach a consensus within this window after the main work is done. ────────── How to confirm the exact cause To see exactly what timeout was applied to your failed transaction, run:
Copy code
sql
-- Replace <txn_id> with the ID from your error message
SHOW TRANSACTION WHERE id = <txn_id>;
Look at the
TimeoutMs
column in the result. * If
TimeoutMs
is
900000
, then something (likely a session variable) explicitly set a 15-minute limit. * If
TimeoutMs
is
3600000
(1 hour) but it still failed early, the issue is likely a BE-side RPC timeout or a network/proxy interruption. Summary of what to change If you are using JDBC, it is best practice to set the session variables explicitly before running the insert:
Copy code
sql
SET query_timeout = 3600;
SET insert_timeout = 3600; -- If on v3.4+
INSERT INTO ...;
-- References -- https://docs.starrocks.io/docs/sql-reference/System_variable -- https://docs.starrocks.io/docs/administration/management/FE_configuration ``` ## References - [administration/management/FE_configuration.md](https://docs.starrocks.io/docs/administration/management/FE_configuration) - [administration/management/BE_configuration.md](https://docs.starrocks.io/docs/administration/management/BE_configuration) - [faq/loading/Loading_faq.md](https://docs.starrocks.io/docs/faq/loading/Loading_faq)
r
@Rocky, if I have an
INSERT... SELECT
, are both
insert_timeout
and
query_timeout
going to impact my query?
r
Whether both impact your query depends entirely on which StarRocks version you are running. In short: They do not both impact the query at the same time. One takes precedence over the other depending on your version. 1. If you are on v3.4.0 or newer Starting with version 3.4.0, StarRocks explicitly separated these two variables to prevent long-running data loads from being killed by short query timeouts. *
insert_timeout
: This is the only variable that governs
INSERT... SELECT
. It covers the entire duration of the operation (the
SELECT
part and the actual writing/committing of data). *
query_timeout
: This variable is ignored for
INSERT
,
UPDATE
,
DELETE
, and
CTAS
statements. It now only applies to read-only
SELECT
statements. 2. If you are on a version older than v3.4.0 In older versions,
insert_timeout
did not exist as a session variable. *
query_timeout
: This was the primary variable governing the execution time of the
INSERT... SELECT
statement. *
insert_load_default_timeout_second
(FE Config)
: This defined the transaction's maximum life in the Transaction Manager. In these older versions, your query would fail if either the
query_timeout
was reached (killing the execution) or the transaction manager's timeout was reached. ────────── Why this matters for your
INSERT... SELECT
Because you previously saw the suggestion for
insert_timeout
in your error message, you are almost certainly on v3.4.0 or later. For your scenario: * Use
insert_timeout
to control the 15-minute failure you are seeing. * Ignore
query_timeout
for this specific task, as it no longer restricts
INSERT
operations in your version. Recommended Setting: If your JDBC app is doing a heavy
INSERT... SELECT
, set this at the start of your session:
Copy code
sql
SET insert_timeout = 3600; -- Set to 1 hour
INSERT INTO target_table SELECT ... FROM source_table;
One Final Catch:
streaming_load_rpc_max_alive_time_sec
Even if
insert_timeout
is set to 1 hour, if the backends (BE) take more than 20 minutes (default) to process the data chunks, they might stop responding to the FE. If your task consistently fails around the 20-minute mark, you may also need to increase
streaming_load_rpc_max_alive_time_sec
in your be.conf. Referencessql-reference/System_variable.mdfaq/loading/Insert_into_faq.mdloading/InsertInto.mdadministration/management/BE_configuration.md Referencesfaq/loading/Insert_into_faq.mdsql-reference/System_variable.mdloading/InsertInto.mdloading/loading_introduction/loading_considerations.mdfaq/resource_isolation_faq.mdadministration/management/FE_configuration.mdsql-reference/sql-statements/loading_unloading/INSERT.md
r
@Rocky, we're definitely on the latest 3.* version.
insert_timeout
is set to
14400
.
insert_load_default_timeout_second
is set to
3600
.
streaming_load_rpc_max_alive_time_sec
is set to
1200
. On my most recent attempt, some batches timed out after only 10 minutes.
r
Since your timeout occurred at exactly 10 minutes (600 seconds), the most likely culprit is a specific internal StarRocks timeout that defaults to exactly that value. 1. The 10-Minute Internal Timeout (
tablet_writer_open_rpc_timeout_sec
)
There is a configuration in the FE (Frontend) called
tablet_writer_open_rpc_timeout_sec
. * Default value: 600 seconds (10 minutes). * What it does: When an
INSERT
starts, the FE sends an RPC (Remote Procedure Call) to the BEs to "open" the tablet writers to prepare for data. If the cluster is under heavy load, has many tablets, or the schema is very complex, this "open" phase can take a long time. * Solution: Increase this in your
fe.conf
(requires FE restart) or check if it can be set via `ADMIN SET FRONTEND CONFIG`:
Copy code
sql
    ADMIN SET FRONTEND CONFIG ("tablet_writer_open_rpc_timeout_sec" = "1200");
2. The Network/Proxy Timeout If you are connecting via a Load Balancer (F5, HAProxy, Nginx) or a Cloud Gateway (AWS ALB/NLB), 10 minutes (600s) is a very common default idle timeout. * If the StarRocks FE is busy processing the
SELECT
part of your
INSERT...SELECT
and doesn't send any data back to the JDBC client for 10 minutes, the proxy may drop the connection. * Check: Does the error message come from the MySQL/StarRocks driver, or does it look like a "Connection reset" or "Communication link failure"? 3. JDBC Socket Timeout Check your JDBC connection string. If you have a
socketTimeout
or
connectTimeout
parameter set, it will override any server-side settings. * Example:
jdbc:<mysql://host:9030/db?socketTimeout=600000>
(600,000ms = 10 minutes). 4.
query_timeout
(Version Verification)
You mentioned you are on "the latest 3.* version." * If you are on v3.3.x, the
insert_timeout
variable does not exist yet. In v3.3,
INSERT
still follows
query_timeout
(default 300s). * If you are on v3.4.x,
insert_timeout
is the correct variable. * Verification: Run
SELECT VERSION();
. If it is
3.3.x
, you must increase
query_timeout
. Recommended Next Steps: 1. Check FE Logs: Look at
fe.log
or
fe.warn.log
for the specific Transaction ID. It will usually state exactly which timeout was triggered (e.g.,
"type:TIMEOUT, msg:tablet writer open timeout"
). 2. Verify the Transaction: Immediately after a failure, run:
Copy code
sql
    SHOW TRANSACTION WHERE id = <txn_id>;
Look at the
TimeoutMs
column. If it says
600000
, the 10-minute limit is being enforced by the FE configuration. Referencesadministration/management/FE_configuration.mdfaq/loading/Insert_into_faq.mdsql-reference/sql-statements/administration/ADMIN_SET_CONFIG.mdloading/InsertInto.md Into)
r
@Rocky, it wasn't "exactly" 10 minutes. They failed when attempting to commit after about 10 minutes. An explicit
commit
fails and gives that exception so it could have happened at any point. Meanwhile, other transactions seem to have finished successfully after about 12 minutes. I don't seem to have easy access to front end logs. Is there a way to just view recent transactions from within a StarRocks prompt?
r
To view recent transaction history directly from your StarRocks prompt, use the
SHOW PROC
system and the
SHOW LOAD
command. Since you don't have FE log access, these are your best tools for identifying failed transaction IDs and the reasons for their failure. 1. How to find Transaction IDs from the prompt Option A:
SHOW PROC
(The "Internal" View)
StarRocks keeps a list of recent transactions in its internal process directory. 1. First, find your Database ID:
Copy code
sql
    SHOW PROC '/transactions';
2. Then, use that ID (e.g.,
1001
) to list finished transactions:
Copy code
sql
    -- Replace 1001 with your DbId from the step above
    SHOW PROC '/transactions/1001/finished';
*This will return a list including
TxnId
,
Label
,
Status
, and
Reason
(which often contains the specific timeout error).* Option B:
SHOW LOAD
(The "Job" View)
Since
INSERT
statements are treated as load jobs, they often appear here.
Copy code
sql
SHOW LOAD ORDER BY CreateTime DESC LIMIT 10;
Look for labels starting with
insert_
. The
ErrorMsg
column here is usually the most readable source for why a commit failed. ────────── 2. Why "About 10 Minutes"? Since your
insert_timeout
and
insert_load_default_timeout
are set much higher, but you are failing at ~10 minutes, check these three specific "hidden" configs: 1.
tablet_writer_open_rpc_timeout_sec
(Default: 300-600s)
If you have a high number of tablets or a complex schema, the "opening" phase of the transaction (before data even flows) can time out. In some versions, the effective timeout is the minimum of this variable and half of your total load timeout. * Check/Set:
ADMIN SET FRONTEND CONFIG ("tablet_writer_open_rpc_timeout_sec" = "1200");
2.
external_table_commit_timeout_ms
(Default: 10s)
Only relevant if you are inserting into an External Table (e.g., Hive, Iceberg). If the metadata operation on the external catalog takes too long during the
COMMIT
phase, it will fail. * Check/Set:
ADMIN SET FRONTEND CONFIG ("external_table_commit_timeout_ms" = "600000");
(10 minutes). 3. JDBC Driver
net_write_timeout
/
net_read_timeout
If the StarRocks FE is working hard to process the commit and doesn't "speak" back to the JDBC driver for a long window, the MySQL protocol itself can time out. * Solution: Try adding
?socketTimeout=1200000
to your JDBC connection string to ensure the client doesn't give up before the server does. Next Step Recommendation Run
SHOW PROC '/transactions/<db_id>/finished';
and look at the
Reason
column for one of the failed rows. If it says "tablet writer open timeout" or "publish version timeout", you'll know exactly which internal RPC reached its limit. -- References -- https://docs.starrocks.io/docs/sql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC -- https://docs.starrocks.io/docs/sql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION -- https://docs.starrocks.io/docs/administration/management/FE_configuration Referencessql-reference/sql-statements/cluster-management/nodes_processes/SHOW_PROC.mdsql-reference/sql-statements/loading_unloading/SHOW_TRANSACTION.mdadministration/management/FE_configuration.mdloading/SQL_transaction.mdsql-reference/information_schema/be_txns.mdadministration/management/BE_configuration.mdusing_starrocks/running_queries.md
r
@Rocky, I found it:
Copy code
TransactionId: 2116006
             Label: 019e85c6-7e05-77f1-a165-bbdf7cefe4ef
       Coordinator: FE: starrocks-shared-data-cluster-fe-1.starrocks-shared-data-cluster-fe-search.starrocks.svc.cluster.local
 TransactionStatus: ABORTED
 LoadJobSourceType: INSERT_STREAMING
       PrepareTime: 2026-06-02 00:40:38
      PreparedTime: NULL
        CommitTime: NULL
       PublishTime: NULL
        FinishTime: 2026-06-02 00:45:43
            Reason: timeout by txn manager
ErrorReplicasCount: 0
        ListenerId: [2038642, 2038673, 2038700, 2038744, 2038765, 2038826, 2038852, 2038894, 2038937, 2039007, 2039062, 2039093, 2039143, 2039192, 2039234, 2039286, 2039347, 2039400, 2039458, 2039496, 2039543, 2039595, 2039642, 2039688, 2039748, 2039788, 2039847, 2039880, 2040035, 2040098, 2040169, 2040223, 2040287, 2040346, 2040429, 2040469, 2040517, 2040570, 2040619, 2040677, 2040724, 2040789, 2040844, 2040900, 2040957, 2041018, 2041075, 2041121, 2041154, 2041185, 2041225, 2041268, 2041301]
         TimeoutMs: 300000
 PreparedTimeoutMs: 86400000
            ErrMsg:
Now the question is why would it time out after 300000ms?
r
The value
300000
in your
TimeoutMs
column is exactly 300 seconds (5 minutes). This is the default value for the
query_timeout
session variable. The fact that your transaction shows
300000
instead of
14400000
(your
insert_timeout
) or
3600000
(your
insert_load_default_timeout_second
) tells us that StarRocks is ignoring your new settings and falling back to the default query timeout. Here is why this is happening and how to fix it: 1. You might be on a version older than v3.4.0 Even if you are on a "late 3.*" version, if it is v3.3.x or lower, the
insert_timeout
variable does not exist or is not honored for
INSERT
jobs. In these versions,
query_timeout
is the only variable that controls the execution time of an
INSERT... SELECT
. * Fix: Run
SET query_timeout = 14400;
before your insert, or upgrade to v3.4.0+. 2. Session Variable Scope Issues Session variables like
insert_timeout
or
query_timeout
only apply to the current database connection. * If your JDBC application is using a Connection Pool (like HikariCP or Druid), the session where you executed
SET insert_timeout = 14400
might not be the same session that actually executes the
INSERT
or the
COMMIT
. * Fix: Ensure your application sets the variable as part of the "connection init SQL" or explicitly sets it globally:
SET GLOBAL insert_timeout = 14400;
(Note: Global settings only apply to new connections created after the change). 3. Explicit Transactions (
BEGIN ... COMMIT
)
If you are using an explicit transaction block:
Copy code
sql
BEGIN;
INSERT INTO ...;
COMMIT;
The timeout for the transaction is often inherited from the session's
query_timeout
at the moment the transaction starts. If the version is slightly older, the transaction manager defaults to 300s unless explicitly told otherwise via session variables it recognizes. 4.
INSERT_STREAMING
Specifics
Your
LoadJobSourceType
is
INSERT_STREAMING
. This usually indicates a batch insert via the MySQL protocol (common in JDBC). * Verify the version:
SELECT VERSION();
* If you are on 3.4.x, check if
insert_timeout
is truly set in the active session:
Copy code
sql
    SHOW VARIABLES LIKE '%timeout%';
* If you see
query_timeout
at
300
and
insert_timeout
at
14400
, but your transaction is still hitting
300000ms
, then your version of StarRocks is likely still using
query_timeout
for the
INSERT_STREAMING
logic. Recommended Fix Try setting both variables to a high value in your session before running the load:
Copy code
sql
SET query_timeout = 14400;
SET insert_timeout = 14400;
-- Then run your INSERT
If the
TimeoutMs
in
SHOW TRANSACTION
still shows
300000
, then the limit is being forced by a global FE configuration. In that case, check the FE's
streaming_load_default_timeout_second
(though its default is usually much higher than 300). Referencessql-reference/System_variable.mdloading/InsertInto.mdfaq/loading/Insert_into_faq.mdsql-reference/sql-statements/loading_unloading/INSERT.md Referencessql-reference/System_variable.mdloading/InsertInto.mdloading/loading_introduction/loading_considerations.mdfaq/loading/Insert_into_faq.mdfaq/operation_maintenance_faq.md