https://pinot.apache.org/ logo
e

Elon

11/19/2020, 10:49 PM
m

Mayank

11/19/2020, 10:52 PM
Is it an expensive query?
e

Elon

11/19/2020, 10:54 PM
But it looks like the error came from a server though
It wasn't expensive, we bounced the servers and the query latency went from 25s to < 1 second after that.
And we saw no red flags, cpu, network, disk usage etc. were fine
Also I wasn't aware those combiner services run on the servers, maybe the logs are messed up in google cloud?
a

Alex

11/19/2020, 11:01 PM
@Mayank yep, error is from the server
m

Mayank

11/19/2020, 11:03 PM
Sorry, my original message was incorrect. this is from server level combine.
👍 1
Was there GC?
e

Elon

11/19/2020, 11:04 PM
I can check
1 sec, also thanks so much for the super quick response!
👍 1
We need to edit the jmx config, looks like we didn't expose that metric
Will do this
m

Mayank

11/20/2020, 1:06 AM
@Elon, just to add a bit more context. The part of the code throwing exception is waiting on worker threads that are executing the query on a bunch of segments.
numBlocksMerged: 0
seems to suggest none of those threads returned within timeout.
a

Alex

11/20/2020, 1:06 AM
@Mayank some context
e

Elon

11/20/2020, 1:07 AM
thanks! So what would likely cause that?
m

Mayank

11/20/2020, 1:07 AM
It is typically things like expensive query, GC, VM freeze
👍 1
a

Alex

11/20/2020, 1:08 AM
our pinot cluster stalled in prod, with 3 out of 5 nodes having those log messages. Their appearance correlate with the time observed query latencies went up
rolling restart fixed the problem for now, but we don't have a root cause
m

Mayank

11/20/2020, 1:08 AM
This is the most common stack trace you will see in case of latency spikes. This is not root cause, but side effect of some other problem.
a

Alex

11/20/2020, 1:09 AM
hmmm
what can cause an issue that is treated by restart?
m

Mayank

11/20/2020, 1:10 AM
GC, High read load
e

Elon

11/20/2020, 1:10 AM
We're using jdk8 - do you recommend us changing to g1gc? Right now it's using concurrent mark sweep.
m

Mayank

11/20/2020, 1:10 AM
Yes, please move to g1gc
❤️ 1
e

Elon

11/20/2020, 1:10 AM
Will do!
m

Mayank

11/20/2020, 1:10 AM
we have not used CMS in Pinot production at all at LinkedIn
Are these realtime nodes?
e

Elon

11/20/2020, 1:11 AM
Yep
m

Mayank

11/20/2020, 1:11 AM
Are you using all the realtime optimizations (like offheap etc)?
a

Alex

11/20/2020, 1:11 AM
they are hybrid
realtime + offline
e

Elon

11/20/2020, 1:11 AM
I believe so, mmap for segments, is that how you do off heap?
Any jvm options you recommend?
m

Mayank

11/20/2020, 1:12 AM
how much data per node?
most of the time we use xms=xmx=16G
e

Elon

11/20/2020, 1:14 AM
6 nodes x 100gb-280gb (for the older nodes w more realtime data)
m

Mayank

11/20/2020, 1:15 AM
ingestion rate?
e

Elon

11/20/2020, 1:15 AM
and ~27gb memory
m

Mayank

11/20/2020, 1:15 AM
ok, you likely need more heap than 16G for sure
e

Elon

11/20/2020, 1:16 AM
But if we' re using mmap for segments that would imply we need to leave space for offheap, right?
m

Mayank

11/20/2020, 1:16 AM
also something like:
<value>-XX:MaxGCPauseMillis=20</value>
👍 2
e

Elon

11/20/2020, 1:17 AM
and how do we take advantage of offheap, is that just the segment loadmode = mmap?
m

Mayank

11/20/2020, 1:17 AM
Copy code
<value>-XX:+UseG1GC</value>
          <value>-XX:+ParallelRefProcEnabled</value>
          <value>-XX:+DisableExplicitGC</value>
I think there's another setting for mmap. Let me find.
e

Elon

11/20/2020, 1:18 AM
thanks!
we ingest ~6-8mb/s but that will be increasing
a

Alex

11/20/2020, 1:19 AM
do we need to set MaxDirectMemory?
e

Elon

11/20/2020, 1:19 AM
@Mayank, you're really help us, thanks!
👍 1
m

Mayank

11/20/2020, 1:19 AM
@Alex it is not needed, but in practice, we set it to size of main-memory - xmx
a

Alex

11/20/2020, 1:21 AM
got it, thank you! do you do it to allocate additional ram? or just historical?
e

Elon

11/20/2020, 1:27 AM
sorry, just want to make sure we know how to take advantage of offheap, I thought it was just setting the load mode to MMAP, lmk if there's anything else
m

Mayank

11/20/2020, 1:27 AM
@Elon ^^