Hello guys, anyone knows how to trobleshoot the following issue? Every single transaction I send is timing out, I can run show commands, but no commands to the data are working, I had restarted FE nodes, but nothing changed.

@Rocky Hello guys, anyone knows how to trobleshoot the following issue? Every single transaction I send is timing out, I can run show commands, but no commands to the data are working, I had restarted FE nodes, but nothing changed.

This sounds like a cluster-wide issue where the Frontend (FE) is unable to communicate with or coordinate with the Backends (BEs) for data-related operations, even though metadata (SHOW commands) is still working. Since restarting FEs didn't work, here is a step-by-step troubleshooting guide to identify the root cause: 1. Check Backend Health First, verify that the BEs are actually alive and reporting to the FE.

sql
SHOW BACKENDS;

* Alive: Are all BEs

true

? If

false

, check

be/log/be.INFO

for crash loops. * UsedCapacity: Check if any BE disk is near 95%. StarRocks has a "storage flood stage" (default 95%) where it rejects all new transactions and load jobs to prevent disk corruption. 2. Check for Metadata Locks or Deadlocks If a specific operation (like a massive

DROP TABLE

or a schema change) is stuck, it can hold a global lock that prevents other transactions from starting. * Run

SHOW PROCESSLIST;

: Look for any query in a

Pending

or

Executing

state for a very long time. * Check FE Logs: Look at

fe/log/fe.warn.log

for any "lock" or "deadlock" messages. * Jstack: If you suspect a deadlock, run

jstack -l <FE_PID> > fe_stack.txt

on the Leader FE and search for

parking to wait for

or

Locked ownable synchronizers

. 3. Check the Transaction Manager If the Global Transaction Manager is overwhelmed or stuck: * Run

SHOW PROC '/transactions';

to see the count of active transactions. If there are thousands of transactions in

PREPARE

or

BEGIN

state, it might be hitting a limit. * Check if there are any stuck Load jobs:

SHOW LOAD WHERE State = 'LOADING';

. A stuck load can sometimes prevent the transaction ID from advancing smoothly. 4. Verify Network/RPC Connectivity Commands to data (SELECT/INSERT) require RPC communication between FE and BE. * Check Ports: Ensure the FE can reach the BEs on ports 9060 (BE RPC port) and 9050 (BE Heartbeat port). * FE Logs: Look for

RpcException

or

thrift_rpc_timeout

in

fe.warn.log

. * BE Logs: Look for

failed to commit transaction

or

timeout

in

be/log/be.WARNING

. 5. Check Disk Space on FEs Metadata commands work from memory, but new transactions must be written to the BDBJE (metadata) log. * Check the disk space on the FE nodes (

df -h

). If the FE metadata partition is 100% full, no new transactions can be recorded. 6. Clock Skew StarRocks is sensitive to time differences between nodes. * Run

date

on all FE and BE nodes. If the difference is more than 5 seconds, it can cause transaction timeouts or heartbeat failures. Recommended Actions: 1. Check

fe.warn.log

immediately for the specific error message associated with the timeout. It will usually tell you if it's a

GlobalTransactionMgr

error or a

Master Not Found

error. 2. Verify Disk Usage on all nodes (both FE and BE). 3. Check BE Logs: If the FE is fine, the BE might be stuck in a long GC pause or a compaction storm. If you find a specific error message in the logs, please share it and I can provide more specific help! References • loading/InsertInto.md • faq/operation_maintenance_faq.md • sql-reference/System_variable.md • administration/management/resource_management/Query_management.md