Hello, team.

2025-12-10 02:20:53.853Z WARN (thrift-server-pool-1520947|1943386) [LockManager.logSlowLockTrace():426] LockManager detects slow lock : {"rid":122933773,"owners":[{"id":1943364,"name":"thrift-server-pool-1520938","type":"INTENTION_EXCLUSIVE","heldFor":8632,"waitTime":0,"stack":["java.base@17.0.17/java.lang.Object.wait(Native Method)","app//com.starrocks.common.util.concurrent.lock.LockManager.lock(LockManager.java:157)","app//com.starrocks.common.util.concurrent.lock.Locker.lock(Locker.java:93)","app//com.starrocks.common.util.concurrent.lock.Locker.tryLockTablesWithIntensiveDbLock(Locker.java:333)","app//com.starrocks.transaction.GlobalTransactionMgr.commitTransactionUnderDatabaseWLock(GlobalTransactionMgr.java:492)","app//com.starrocks.transaction.GlobalTransactionMgr.retryCommitOnRateLimitExceeded(GlobalTransactionMgr.java:454)","app//com.starrocks.transaction.GlobalTransactionMgr.commitAndPublishTransaction(GlobalTransactionMgr.java:422)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommitImpl(FrontendServiceImpl.java:1323)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommit(FrontendServiceImpl.java:1276)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:6021)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:5998)","app//org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)","app//org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)","app//com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.17/java.lang.Thread.run(Thread.java:840)"]},{"id":1942257,"name":"pool-25-thread-5162","type":"INTENTION_SHARED","heldFor":4229,"waitTime":0,"stack":["java.base@17.0.17/java.lang.Object.wait(Native Method)","app//com.starrocks.common.util.concurrent.lock.LockManager.lock(LockManager.java:155)","app//com.starrocks.common.util.concurrent.lock.Locker.lock(Locker.java:93)","app//com.starrocks.common.util.concurrent.lock.Locker.lockTablesWithIntensiveDbLock(Locker.java:300)","app//com.starrocks.load.routineload.RoutineLoadJob.plan(RoutineLoadJob.java:905)","app//com.starrocks.load.routineload.KafkaTaskInfo.plan(KafkaTaskInfo.java:242)","app//com.starrocks.load.routineload.KafkaTaskInfo.createRoutineLoadTask(KafkaTaskInfo.java:196)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler.scheduleOneTask(RoutineLoadTaskScheduler.java:245)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler.lambda$submitToSchedule$1(RoutineLoadTaskScheduler.java:168)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler$$Lambda$3122/0x000000f002050b10.run(Unknown Source)","java.base@17.0.17/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)","java.base@17.0.17/java.util.concurrent.FutureTask.run(FutureTask.java:264)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.17/java.lang.Thread.run(Thread.java:840)"]},{"id":1942437,"name":"thrift-server-pool-1520656","type":"INTENTION_EXCLUSIVE","heldFor":8254,"waitTime":0,"stack":["java.base@17.0.17/java.lang.Object.wait(Native Method)","app//com.starrocks.common.util.concurrent.lock.LockManager.lock(LockManager.java:157)","app//com.starrocks.common.util.concurrent.lock.Locker.lock(Locker.java:93)","app//com.starrocks.common.util.concurrent.lock.Locker.tryLockTablesWithIntensiveDbLock(Locker.java:333)","app//com.starrocks.transaction.GlobalTransactionMgr.commitTransactionUnderDatabaseWLock(GlobalTransactionMgr.java:492)","app//com.starrocks.transaction.GlobalTransactionMgr.retryCommitOnRateLimitExceeded(GlobalTransactionMgr.java:454)","app//com.starrocks.transaction.GlobalTransactionMgr.commitAndPublishTransaction(GlobalTransactionMgr.java:422)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommitImpl(FrontendServiceImpl.java:1323)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommit(FrontendServiceImpl.java:1276)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:6021)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:5998)","app//org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)","app//org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)","app//com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.17/java.lang.Thread.run(Thread.java:840)"]},{"id":1941765,"name":"pool-25-thread-5160","type":"INTENTION_SHARED","heldFor":3940,"waitTime":0,"stack":["java.base@17.0.17/java.lang.Object.wait(Native Method)","app//com.starrocks.common.util.concurrent.lock.LockManager.lock(LockManager.java:155)","app//com.starrocks.common.util.concurrent.lock.Locker.lock(Locker.java:93)","app//com.starrocks.common.util.concurrent.lock.Locker.lockTablesWithIntensiveDbLock(Locker.java:300)","app//com.starrocks.load.routineload.RoutineLoadJob.plan(RoutineLoadJob.java:905)","app//com.starrocks.load.routineload.KafkaTaskInfo.plan(KafkaTaskInfo.java:242)","app//com.starrocks.load.routineload.KafkaTaskInfo.createRoutineLoadTask(KafkaTaskInfo.java:196)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler.scheduleOneTask(RoutineLoadTaskScheduler.java:245)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler.lambda$submitToSchedule$1(RoutineLoadTaskScheduler.java:168)","app//com.starrocks.load.routineload.RoutineLoadTaskScheduler$$Lambda$3122/0x000000f002050b10.run(Unknown Source)","java.base@17.0.17/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)","java.base@17.0.17/java.util.concurrent.FutureTask.run(FutureTask.java:264)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.17/java.lang.Thread.run(Thread.java:840)"]},{"id":1941454,"name":"thrift-server-pool-1520312","type":"INTENTION_EXCLUSIVE","heldFor":8757,"waitTime":0,"stack":["java.base@17.0.17/jdk.internal.misc.Unsafe.park(Native Method)","java.base@17.0.17/java.util.concurrent.locks.LockSupport.park(LockSupport.java:211)","java.base@17.0.17/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:715)","java.base@17.0.17/java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:938)","java.base@17.0.17/java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:959)","app//com.starrocks.load.routineload.RoutineLoadJob.writeLock(RoutineLoadJob.java:491)","app//com.starrocks.load.routineload.RoutineLoadJob.executeBeforeCheck(RoutineLoadJob.java:986)","app//com.starrocks.load.routineload.RoutineLoadJob.beforeCommitted(RoutineLoadJob.java:976)","app//com.starrocks.transaction.TransactionState.beforeStateTransform(TransactionState.java:687)","app//com.starrocks.transaction.DatabaseTransactionMgr.commitPreparedTransaction(DatabaseTransactionMgr.java:509)","app//com.starrocks.transaction.DatabaseTransactionMgr.commitTransaction(DatabaseTransactionMgr.java:570)","app//com.starrocks.transaction.GlobalTransactionMgr.commitTransaction(GlobalTransactionMgr.java:286)","app//com.starrocks.transaction.GlobalTransactionMgr.commitTransactionUnderDatabaseWLock(GlobalTransactionMgr.java:497)","app//com.starrocks.transaction.GlobalTransactionMgr.retryCommitOnRateLimitExceeded(GlobalTransactionMgr.java:454)","app//com.starrocks.transaction.GlobalTransactionMgr.commitAndPublishTransaction(GlobalTransactionMgr.java:422)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommitImpl(FrontendServiceImpl.java:1323)","app//com.starrocks.service.FrontendServiceImpl.loadTxnCommit(FrontendServiceImpl.java:1276)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:6021)","app//com.starrocks.thrift.FrontendService$Processor$loadTxnCommit.getResult(FrontendService.java:5998)","app//org.apache.thrift.ProcessFunction.process(ProcessFunction.java:40)","app//org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:40)","app//com.starrocks.common.SRTThreadPoolServer$WorkerProcess.run(SRTThreadPoolServer.java:311)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)","java.base@17.0.17/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)","java.base@17.0.17/java.lang.Thread.run(Thread.java:840)"]},{"id":1943141,"name":"thrift-server-pool-1520881","type":"INTENTION_SHARED","heldFor":5484,"waitTime":0,"stack":["java.base@17.0.17/java.lang.Object.wait(Native Method)","app//com.starrocks.common.util.concurrent.lock.LockManager.lock(LockManager.java:155)","app//com.starrocks.common.util.concurrent.lock.Locker.lock(Locker.java:93)","app//com.starrocks.common.util.concurrent.lock.Locker.lockTablesWithIntensiveDbLock(Locker.java:300)","app//com.starrocks.common.util.concurrent.lock.AutoCloseableLock.<init>

We are experiencing issues where query latency occasionally increases even in situations with low concurrency due to the slow locking of routine loads. Do you have any ideas?

which version are you on?

@Kevin Cai v3.5.9

is there a deadlock there when retrieving jstack of FE process with

jstack -l <fe_pid>

maybe system is automatically recovered. But cause of the locks, sometimes query latency increased.

ok. so it is just slow, not dead lock.

how can I improve the slow locks for routine load? increasing of

max_batch_interval,

max_batch_rows

?