Channels
#troubleshooting
Title
e
Elon
11/19/2020, 10:49 PMm
Mayank
11/19/2020, 10:52 PM
Is it an expensive query?
e
Elon
11/19/2020, 10:54 PMBut it looks like the error came from a server though
It wasn't expensive, we bounced the servers and the query latency went from 25s to < 1 second after that.
And we saw no red flags, cpu, network, disk usage etc. were fine
Also I wasn't aware those combiner services run on the servers, maybe the logs are messed up in google cloud?
a
Alex
11/19/2020, 11:01 PM@Mayank yep, error is from the server
m
Mayank
11/19/2020, 11:03 PM
Sorry, my original message was incorrect. this is from server level combine.
👍 1
Was there GC?
e
Elon
11/19/2020, 11:04 PMI can check
1 sec, also thanks so much for the super quick response!
👍 1
We need to edit the jmx config, looks like we didn't expose that metric
Will do this
m
Mayank
11/20/2020, 1:06 AM
@Elon, just to add a bit more context. The part of the code throwing exception is waiting on worker threads that are executing the query on a bunch of segments.
numBlocksMerged: 0a
Alex
11/20/2020, 1:06 AM@Mayank some context
e
Elon
11/20/2020, 1:07 AMthanks! So what would likely cause that?
m
Mayank
11/20/2020, 1:07 AM
It is typically things like expensive query, GC, VM freeze
👍 1
a
Alex
11/20/2020, 1:08 AMour pinot cluster stalled in prod, with 3 out of 5 nodes having those log messages. Their appearance correlate with the time observed query latencies went up
rolling restart fixed the problem for now, but we don't have a root cause
m
Mayank
11/20/2020, 1:08 AM
This is the most common stack trace you will see in case of latency spikes. This is not root cause, but side effect of some other problem.
a
Alex
11/20/2020, 1:09 AMhmmm
what can cause an issue that is treated by restart?
m
Mayank
11/20/2020, 1:10 AM
GC, High read load
e
Elon
11/20/2020, 1:10 AMWe're using jdk8 - do you recommend us changing to g1gc? Right now it's using concurrent mark sweep.
m
Mayank
11/20/2020, 1:10 AM
Yes, please move to g1gc
❤️ 1
e
Elon
11/20/2020, 1:10 AMWill do!
m
Mayank
11/20/2020, 1:10 AM
we have not used CMS in Pinot production at all at LinkedIn
Are these realtime nodes?
e
Elon
11/20/2020, 1:11 AMYep
m
Mayank
11/20/2020, 1:11 AM
Are you using all the realtime optimizations (like offheap etc)?
a
Alex
11/20/2020, 1:11 AMthey are hybrid
realtime + offline
e
Elon
11/20/2020, 1:11 AMI believe so, mmap for segments, is that how you do off heap?
Any jvm options you recommend?
m
Mayank
11/20/2020, 1:12 AM
how much data per node?
most of the time we use xms=xmx=16G
e
Elon
11/20/2020, 1:14 AM6 nodes x 100gb-280gb (for the older nodes w more realtime data)
m
Mayank
11/20/2020, 1:15 AM
ingestion rate?
e
Elon
11/20/2020, 1:15 AMand ~27gb memory
m
Mayank
11/20/2020, 1:15 AM
ok, you likely need more heap than 16G for sure
e
Elon
11/20/2020, 1:16 AMBut if we' re using mmap for segments that would imply we need to leave space for offheap, right?
m
Mayank
11/20/2020, 1:16 AM
also something like:
<value>-XX:MaxGCPauseMillis=20</value>👍 2
e
Elon
11/20/2020, 1:17 AMand how do we take advantage of offheap, is that just the segment loadmode = mmap?
m
Mayank
11/20/2020, 1:17 AM
Copy code
<value>-XX:+UseG1GC</value>
<value>-XX:+ParallelRefProcEnabled</value>
<value>-XX:+DisableExplicitGC</value>I think there's another setting for mmap. Let me find.
e
Elon
11/20/2020, 1:18 AMthanks!
we ingest ~6-8mb/s but that will be increasing
a
Alex
11/20/2020, 1:19 AMdo we need to set MaxDirectMemory?
e
Elon
11/20/2020, 1:19 AM@Mayank, you're really help us, thanks!
👍 1
m
Mayank
11/20/2020, 1:19 AM
@Alex it is not needed, but in practice, we set it to size of main-memory - xmx
a
Alex
11/20/2020, 1:21 AMgot it, thank you! do you do it to allocate additional ram? or just historical?
m
Mayank
11/20/2020, 1:27 AM
e
Elon
11/20/2020, 1:27 AMsorry, just want to make sure we know how to take advantage of offheap, I thought it was just setting the load mode to MMAP, lmk if there's anything else
m
Mayank
11/20/2020, 1:27 AM
@Elon ^^