http://starrocks.io logo
Join Slack
Powered by
``` Found one Java-level deadlock: ==============...
# questions-and-troubleshooting
r
Copy code
Found one Java-level deadlock:
=============================
"txnTimeoutChecker":
  waiting for ownable synchronizer 0x0000000649435600, (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync),
  which is held by "thrift-server-pool-3256604"
"thrift-server-pool-3256604":
  waiting for ownable synchronizer 0x0000000635c3ed90, (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync),
  which is held by "txnTimeoutChecker"

Java stack information for the threads listed above:
===================================================
"txnTimeoutChecker":
	at jdk.internal.misc.Unsafe.park(java.base@11.0.23/Native Method)
	- parking to wait for  <0x0000000649435600> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.23/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.23/AbstractQueuedSynchronizer.java:885)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.23/AbstractQueuedSynchronizer.java:917)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.23/AbstractQueuedSynchronizer.java:1240)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.23/ReentrantReadWriteLock.java:959)
	at com.starrocks.transaction.TransactionState.writeLock(TransactionState.java:340)
	at com.starrocks.transaction.DatabaseTransactionMgr.abortTransaction(DatabaseTransactionMgr.java:531)
	at com.starrocks.transaction.DatabaseTransactionMgr.abortTransaction(DatabaseTransactionMgr.java:1384)
	at com.starrocks.transaction.DatabaseTransactionMgr.abortTimeoutTxns(DatabaseTransactionMgr.java:1689)
	at com.starrocks.transaction.GlobalTransactionMgr.abortTimeoutTxns(GlobalTransactionMgr.java:599)
	at com.starrocks.server.GlobalStateMgr$4.runAfterCatalogReady(GlobalStateMgr.java:1703)
	at com.starrocks.common.util.FrontendDaemon.runOneCycle(FrontendDaemon.java:72)
	at com.starrocks.common.util.Daemon.run(Daemon.java:107)
"thrift-server-pool-3256604":
	at jdk.internal.misc.Unsafe.park(java.base@11.0.23/Native Method)
	- parking to wait for  <0x0000000635c3ed90> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(java.base@11.0.23/LockSupport.java:194)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(java.base@11.0.23/AbstractQueuedSynchronizer.java:885)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(java.base@11.0.23/AbstractQueuedSynchronizer.java:917)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(java.base@11.0.23/AbstractQueuedSynchronizer.java:1240)
	at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(java.base@11.0.23/ReentrantReadWriteLock.java:959)
	at com.starrocks.load.routineload.RoutineLoadJob.writeLock(RoutineLoadJob.java:470)
	at com.starrocks.load.routineload.RoutineLoadJob.executeBeforeCheck(RoutineLoadJob.java:953)
	at com.starrocks.load.routineload.RoutineLoadJob.beforeCommitted(RoutineLoadJob.java:943)
	at com.starrocks.transaction.TransactionState.beforeStateTransform(TransactionState.java:569)
	at com.starrocks.transaction.DatabaseTransactionMgr.commitPreparedTransaction(DatabaseTransactionMgr.java:435)
	at com.starrocks.transaction.DatabaseTransactionMgr.commitTransaction(DatabaseTransactionMgr.java:489)
	at com.starrocks.transaction.GlobalTransactionMgr.commitTransaction(GlobalTransactionMgr.java:266)
	at com.starrocks.transaction.GlobalTransactionMgr.commitTransactionUnderDatabaseWLock(GlobalTransactionMgr.java:444)
	at com.starrocks.transaction.GlobalTransactionMgr.retryCommitOnRateLimitExceeded(GlobalTransactionMgr.java:402)
	at com.starrocks.transaction.GlobalTransactionMgr.commitAndPublishTransaction(GlobalTransactionMgr.java:370)
	at com.starrocks.service.FrontendServiceImpl.loadTxnCommitImpl(FrontendServiceImpl.java:1416)
	at com.starrocks.service.FrontendServiceImpl.loadTxnCommit(FrontendServiceImpl.java:1369)
	at com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:5061)
	at com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:5038)
	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)
	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)
	at com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@11.0.23/ThreadPoolExecutor.java:1128)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@11.0.23/ThreadPoolExecutor.java:628)
	at java.lang.Thread.run(java.base@11.0.23/Thread.java:829)

Found 1 deadlock.
Hi team , @Rocky got the above dead lock in fe thread dump , transactions were stuck in PREPARED state and no queries were working Starrocks versions - 3.3.2 , is this some known bug?
r
Hi, I'm Rocky. Based on the stack traces you provided, this is a known deadlock issue in StarRocks version 3.3.2 (and some earlier 3.2.x/3.1.x versions) involving a lock inversion between the Routine Load Job and the Transaction Manager. Deadlock Analysis From your thread dump, we can see two threads competing for locks in a circular way: 1. Thread 
thrift-server-pool-3256604
(The "Committer"): * Holding: 
TransactionState
lock (acquired during 
commitTransaction
). * Waiting for: 
RoutineLoadJob
lock (at 
RoutineLoadJob.writeLock
inside the 
beforeCommitted
callback). * Lock Order: 
TransactionState
-> 
RoutineLoadJob
. 2. Thread 
txnTimeoutChecker
(The "Aborter"): * Holding: 
RoutineLoadJob
lock (likely acquired during Routine Load maintenance or state updates). * Waiting for: 
TransactionState
lock (at 
DatabaseTransactionMgr.abortTransaction
line 531). * Lock Order: 
RoutineLoadJob
-> 
TransactionState
. This circular dependency (
Txn -> Job
vs 
Job -> Txn
) causes both threads to hang. Since 
txnTimeoutChecker
is responsible for cleaning up expired transactions, a deadlock here will cause transactions to remain in the 
PREPARED
state indefinitely, eventually exhausting resources and causing the cluster to stop processing queries. Resolution This issue has been identified and fixed in newer releases by optimizing the locking sequence and reducing the scope of held locks. * Fixed Versions: This is fixed in StarRocks 3.3.3 and later versions (as well as 3.2.10+ and 3.1.13+). * Recommendation: I highly recommend upgrading your cluster to at least 3.3.3 (or the latest stable 3.3.x patch, like 3.3.8 or 3.3.13) to resolve this and other stability issues. Temporary Workaround If you cannot upgrade immediately, you can try: 1. Identify the specific Routine Load job: Use 
SHOW ROUTINE LOAD
to see if any jobs are in a 
PAUSED
or 
STALLED
state. 2. Restart FE: A restart of the leader FE node will release the Java-level locks and clear the deadlock, though it may recur if the same conditions (high load + timeouts) are met. 3. Check Routine Load Config: Ensure your Routine Load tasks aren't timing out too frequently by checking Kafka connectivity and task execution times. Let me know if you need help with the upgrade process! References • knowledge/trouble_shooting/fe_deadlock_problems.md
r
Maybe you should upgrade to 3.5 and up
Open in Slack
PreviousNext
Powered by