Hi Team, Is there a way to increase the queue size...
# troubleshooting
v
Hi Team, Is there a way to increase the queue size as pinot server has build around 10d + lag with so many tasks piled up in queue.
Copy code
2025/06/02 18:28:55.712 INFO [HelixTaskExecutor] [ZkClient-EventThread-119-xxxxxxxxx:2181,xxxxxxxx:2181,xxxxxxx:2181] Submit task: d81d401a-b902-4ca4-bcf9-ade6675e0e44 to pool: java.util.concurrent.ThreadPoolExecutor@1339f332[Running, pool size = 40, active threads = 40, queued tasks = 1373, completed tasks = 10622]
m
Are you referring to minion tasks? How many minions do you have?
v
We dont have minions role. This is what i see in pinot server helix logs.
m
So these are state transitions, and the lag is for ingestion?
This usually means you are under provisioned
x
I think those are segments online/offline events?
m
How’s the cpu/memory usage on controller and server? We need to find the choke point here
v
We have 32 pinot server with 35TB disks 1TB RAM, 80 core on each server. resourcing part we havent utilised 10% yet. Looks ok on provision part. Yes, we see cont these messages of segment moving from OFFLINE to Consuming.
Copy code
2025/06/02 18:28:55.712 INFO [HelixTaskExecutor] [ZkClient-EventThread-119-xxxxxxx:2181,xxxxxxxx:2181,x:2181] Scheduling message d81dxxxxxxx401a-b902-4ca4-bcf9-ade6675e0e44:123_test_REALTIME:123_test__9__1260__20250602T1258Z, OFFLINE->CONSUMING
Controller max heap is 18GB, configured is 24GB
m
Ok, then the bottleneck is elsewhere. How often are segments being committed
v
segments are getting committed every hour
m
Then that doesn’t explain the backlog at all.
v
Ack. the restart fixed the issue and lag cleared. The moment before the lag started i could see connection break with ZK, maybe that caused the issue.