https://pinot.apache.org/ logo
Join Slack
Powered by
Hello All, I am having an issue with my Pinot clus...
# troubleshooting
d
Hello All, I am having an issue with my Pinot cluster. Pinot 1.4.0: server heap baseline keeps climbing; timed-out multi-stage (MSE) queries leave leaf stages that never release memory On our 3-node cluster (Pinot 1.4.0, 52 GB server heap, replication 2), the server's live heap after GC climbs steadily, from ~10 GB after a restart to 42–48 GB within ~3 weeks. Only a restart brings it back down. Before that point the heap accountant's panic level starts killing all queries, and one server eventually died with 
OutOfMemoryError: Java heap space
. A 
jmap -histo:live
shows the live heap is mostly 
String
/ 
Object[]
/ 
Long
(~38 GB). A thread dump with no queries running shows thousands of parked MSE threads, the oldest ~21 days old: • ~400 threads in 
LeafOperator.addResultsBlock
→ 
ArrayBlockingQueue.offer
, each holding a full queue of `SelectionResultsBlock`s (~100 blocks, ~90–120 MB each); • ~3,700 threads in 
SortedMailboxReceiveOperator
/ 
MailboxReceiveOperator
waiting on child stages. We can reproduce it on a single-node setup: every MSE JOIN query that times out on the broker leaves one leaf thread parked permanently (still there after 5.5 h, for queries with a 1.5–2 s timeout), holding its buffered rows. A query that completes leaves nothing behind. 
pinot.broker.enable.query.cancellation=true
doesn't change this. 
SET maxStreamingPendingBlocks=10
shrinks each leak ~10× but doesn't stop it. Is this a known issue, or fixed after 1.4.0? Happy to share thread dumps, histograms and the reproduction script. Thankyou in Advance.
m
Can you reproduce with 1.5? cc @Gonzalo Ortiz @Yash Mayya
g
I’m taking a look. 1.4 is more than 1 year old and we since then found a couple of cancellation issues when queries failed, so the issue will probably be fixed in 1.5 and 1.6 (in RC right now)
I found the issue. As expected, this is a known problem in 1.4.0, and it is fixed in 1.5.0. There are two bugs that combine. First, in 1.4.0 the broker sends the absolute passive deadline (an epoch timestamp) in the 
extraPassiveTimeoutMs
field, and the server adds it to the query timeout as if it were a duration. So every MSE wait on the server (leaf queue offer/poll, mailbox receive and send) gets a deadline about 55 years in the future and never times out. That one was fixed in github.com/apache/pinot/pull/16530. Second, when a leaf stage is cancelled, 
LeafOperator
interrupts the SSE thread once. That thread then tries to add the final “end of stream” block in a finally block. The queue is full and nobody reads it anymore, so the thread parks there forever with the whole queue of 
SelectionResultsBlocks
. That is why 
maxStreamingPendingBlocks=10
makes each leak ~10x smaller without stopping it, and why enabling query cancellation does not help. That was fixed in github.com/apache/pinot/pull/16582 (plus a related cancellation fix in github.com/apache/pinot/pull/16625). The real fix is to upgrade to 1.5.x. As a stopgap on 1.4.0, you can try adding 
SET extraPassiveTimeoutMs=1000;
to your MSE queries. The dispatcher copies the query options over the broken value, so the servers get a real deadline back. I haven’t tested this on 1.4.0, so please check it with your reproduction script first.
thanks 1
m
Thanks @Gonzalo Ortiz. @Dhanesh Dhakrey I highly recommend moving to 1.5 since there has been many fixes/improvements into MSE engine since 1.4, and you'd want to avoid uncovering each one at a time.
k
hi,can thread protection shared state and resource management use the "try-with-resources" mechanism to solve the work of regular cleanup? What should I do if error handling fails to generate an alarm in the context? 🙄
Open in Slack
PreviousNext
Powered by