https://pinot.apache.org/ logo
Channels
Powered by
m

Matt

01/26/2021, 10:11 PM
All Pinot Server pods keeps crashing with following error. Anyone came across this before?
Copy code
[Times: user=0.02 sys=0.00, real=0.00 secs]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f104649b6ff, pid=1, tid=0x00007ee665d06700
#
# JRE version: OpenJDK Runtime Environment (8.0_282-b08) (build 1.8.0_282-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.282-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x15c6ff]
#
# Core dump written. Default location: /opt/pinot/core or core.1
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
d

Daniel Lavoie

01/26/2021, 10:17 PM
Can you share some details about your 
Xmx
and available off heap setup?
m

Matt

01/26/2021, 10:26 PM
@Daniel Lavoie
Copy code
jvmOpts: "-Xms512M -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/dev/stdout -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 "
d

Daniel Lavoie

01/26/2021, 10:27 PM
How much ram is ram is available to that box?
m

Matt

01/26/2021, 10:27 PM
Also seeing some WARNs
Copy code
my-pinot-controller-0 controller 2021/01/26 22:17:09.409 WARN [CallbackHandler] [ZkClient-EventThread-29-my-pinot-zk-cp-zookeeper.logging.svc.cluster.local:2181] Callback handler received event in wrong order. Listener: org.apache.helix.controller.GenericHelixController@362617cf, path: /pinot-quickstart/INSTANCES/Server_my-pinot
my-pinot-controller-0 controller 2021/01/26 22:17:12.039 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-quickstart-(83ee1db3_TASK)] Fail to read record for paths: {/pinot-quickstart/INSTANCES/Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098/MESSAGES/51a431e9-c0e1-4e08-9bcb-aee747608526=-101, /pinot-quickstart/INSTANCES/Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098/MESSAGES/f9005198-fe85-4cad-84e8-0df918de95d9=-101
The box has 26G RAM
d

Daniel Lavoie

01/26/2021, 10:28 PM
Server requires 50% heap and 50% non heap
Depending on the segments you have within a server, it’s important to have memory boundaries.
Is it a multi-tenant box?
m

Matt

01/26/2021, 10:30 PM
sort of k8s single tenant , as I restricted with memory boundaries. 26G is fully available to Pinot.
d

Daniel Lavoie

01/26/2021, 10:31 PM
ok, then make sure you have k8s memory request x2 your jvm Xmx
that’s a rule of thumb
for servers of course
Controller and Broker may configure a Xmx that is nearly maxing the k8s memory request
m

Matt

01/26/2021, 10:32 PM
I increased the xmx to 8G now, the mem req is 26G
d

Daniel Lavoie

01/26/2021, 10:32 PM
That’s not what i meant
if you have a 4G XMX, ensure you configure a 
resources.request.memory: 8Gi
to your pod.
By default, if there’s no limit, the pod will think it can use up to 26G of non heap
Until the container runtime says no, no, no
m

Matt

01/26/2021, 10:34 PM
That can happen even if it is 8G right?
d

Daniel Lavoie

01/26/2021, 10:35 PM
You need to think about 2 memory configuration.
The pod is a “VM”, the JVM is running inside it. When working with offheap, the jvm will ask the OS, how much off heap can I use?. If the pod is configured without memory limit. It will tell the JVM that 26G is available.
That 26G will not be reserved
because others pods will also think they can use that.
so, having a pod with a hard 8G limit, will garantee that the JVM will not go over the fence.
m

Matt

01/26/2021, 10:37 PM
ahh ok, I am actually using AWS node with 122GB of RAM and 26 GB is mem request
26GB mem request for single Pinot server
d

Daniel Lavoie

01/26/2021, 10:37 PM
Ok!
then you can even bump it to 12GB xmx if the server pod has 26Gi request
m

Matt

01/26/2021, 10:38 PM
Thanks @Daniel Lavoie, I increased the xmx from 4G to 8G and servers are up
I can use 12G
d

Daniel Lavoie

01/26/2021, 10:38 PM
the rule of thump is ~50%
m

Matt

01/26/2021, 10:39 PM
ok, How abt -Xms12G -Xmx12G
d

Daniel Lavoie

01/26/2021, 10:40 PM
matching Xms is always a good thing in a container environment. that memory is wasted anyway if it is request by the pod
m

Matt

01/26/2021, 10:41 PM
great Thanks for the help here
All the segments are in bad status and search not working. So I have to restore from backup. If the issue was OOM then I assume it might have caused some rouge segments. Now finding those and changing the offset in zookeeper to skip those will be hard..!!!
d

Daniel Lavoie

01/26/2021, 10:55 PM
Are segments being reloaded?
They will remains in bad state until everything is reloaded.
m

Matt

01/26/2021, 10:57 PM
Ohh triggered a reload now , see how it goes
d

Daniel Lavoie

01/26/2021, 10:57 PM
watch your CPU and Disk IO, that’s a good tell of what’s happening
m

Matt

01/26/2021, 11:08 PM
yes both spiked, especially disk
and now all segments came back to good and Pinot is trying to catchup with stream.
catching up very slowly though
d

Daniel Lavoie

01/26/2021, 11:17 PM
cries in 5 hour segments reloads
👍 1
Powered by