https://pinot.apache.org/ logo
Join Slack
Powered by
e
Untitled
Untitled
m
Is it an expensive query?
e
But it looks like the error came from a server though
It wasn't expensive, we bounced the servers and the query latency went from 25s to < 1 second after that.
And we saw no red flags, cpu, network, disk usage etc. were fine
Also I wasn't aware those combiner services run on the servers, maybe the logs are messed up in google cloud?
a
@Mayank yep, error is from the server
m
Sorry, my original message was incorrect. this is from server level combine.
👍 1
Was there GC?
e
I can check
1 sec, also thanks so much for the super quick response!
👍 1
We need to edit the jmx config, looks like we didn't expose that metric
Will do this
m
@Elon, just to add a bit more context. The part of the code throwing exception is waiting on worker threads that are executing the query on a bunch of segments. 
numBlocksMerged: 0
seems to suggest none of those threads returned within timeout.
a
@Mayank some context
e
thanks! So what would likely cause that?
m
It is typically things like expensive query, GC, VM freeze
👍 1
a
our pinot cluster stalled in prod, with 3 out of 5 nodes having those log messages. Their appearance correlate with the time observed query latencies went up
rolling restart fixed the problem for now, but we don't have a root cause
m
This is the most common stack trace you will see in case of latency spikes. This is not root cause, but side effect of some other problem.
a
hmmm
what can cause an issue that is treated by restart?
m
GC, High read load
e
We're using jdk8 - do you recommend us changing to g1gc? Right now it's using concurrent mark sweep.
m
Yes, please move to g1gc
❤️ 1
e
Will do!
m
we have not used CMS in Pinot production at all at LinkedIn
Are these realtime nodes?
e
Yep
m
Are you using all the realtime optimizations (like offheap etc)?
a
they are hybrid
realtime + offline
e
I believe so, mmap for segments, is that how you do off heap?
Any jvm options you recommend?
m
how much data per node?
most of the time we use xms=xmx=16G
e
6 nodes x 100gb-280gb (for the older nodes w more realtime data)
m
ingestion rate?
e
and ~27gb memory
m
ok, you likely need more heap than 16G for sure
e
But if we' re using mmap for segments that would imply we need to leave space for offheap, right?
m
also something like: 
<value>-XX:MaxGCPauseMillis=20</value>
👍 2
e
and how do we take advantage of offheap, is that just the segment loadmode = mmap?
m
Copy code
<value>-XX:+UseG1GC</value>
          <value>-XX:+ParallelRefProcEnabled</value>
          <value>-XX:+DisableExplicitGC</value>
I think there's another setting for mmap. Let me find.
e
thanks!
we ingest ~6-8mb/s but that will be increasing
@Mayank, you're really help us, thanks!
👍 1
a
do we need to set MaxDirectMemory?
m
@Alex it is not needed, but in practice, we set it to size of main-memory - xmx
a
got it, thank you! do you do it to allocate additional ram? or just historical?
e
sorry, just want to make sure we know how to take advantage of offheap, I thought it was just setting the load mode to MMAP, lmk if there's anything else
m
@Elon ^^
Open in Slack
PreviousNext
Powered by