https://pinot.apache.org/ logo
Join Slack
Powered by
All Pinot Server pods keeps crashing with followin...
# troubleshooting
m
All Pinot Server pods keeps crashing with following error. Anyone came across this before?
Copy code
[Times: user=0.02 sys=0.00, real=0.00 secs]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f104649b6ff, pid=1, tid=0x00007ee665d06700
#
# JRE version: OpenJDK Runtime Environment (8.0_282-b08) (build 1.8.0_282-b08)
# Java VM: OpenJDK 64-Bit Server VM (25.282-b08 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# C  [libc.so.6+0x15c6ff]
#
# Core dump written. Default location: /opt/pinot/core or core.1
#
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid1.log
d
Can you share some details about your 
Xmx
and available off heap setup?
m
@Daniel Lavoie
Copy code
jvmOpts: "-Xms512M -Xmx4G -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+PrintGCDetails -XX:+PrintGCDateStamps -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCApplicationConcurrentTime -Xloggc:/dev/stdout -XX:+UnlockExperimentalVMOptions -XX:+UseCGroupMemoryLimitForHeap -XX:MaxRAMFraction=1 "
d
How much ram is ram is available to that box?
m
Also seeing some WARNs
Copy code
my-pinot-controller-0 controller 2021/01/26 22:17:09.409 WARN [CallbackHandler] [ZkClient-EventThread-29-my-pinot-zk-cp-zookeeper.logging.svc.cluster.local:2181] Callback handler received event in wrong order. Listener: org.apache.helix.controller.GenericHelixController@362617cf, path: /pinot-quickstart/INSTANCES/Server_my-pinot
my-pinot-controller-0 controller 2021/01/26 22:17:12.039 WARN [ZkBaseDataAccessor] [HelixController-pipeline-task-pinot-quickstart-(83ee1db3_TASK)] Fail to read record for paths: {/pinot-quickstart/INSTANCES/Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098/MESSAGES/51a431e9-c0e1-4e08-9bcb-aee747608526=-101, /pinot-quickstart/INSTANCES/Server_my-pinot-server-0.my-pinot-server-headless.logging.svc.cluster.local_8098/MESSAGES/f9005198-fe85-4cad-84e8-0df918de95d9=-101
The box has 26G RAM
d
Server requires 50% heap and 50% non heap
Depending on the segments you have within a server, it’s important to have memory boundaries.
Is it a multi-tenant box?
m
sort of k8s single tenant , as I restricted with memory boundaries. 26G is fully available to Pinot.
d
ok, then make sure you have k8s memory request x2 your jvm Xmx
that’s a rule of thumb
for servers of course
Controller and Broker may configure a Xmx that is nearly maxing the k8s memory request
m
I increased the xmx to 8G now, the mem req is 26G
d
That’s not what i meant
if you have a 4G XMX, ensure you configure a 
resources.request.memory: 8Gi
to your pod.
By default, if there’s no limit, the pod will think it can use up to 26G of non heap
Until the container runtime says no, no, no
m
That can happen even if it is 8G right?
d
You need to think about 2 memory configuration.
The pod is a “VM”, the JVM is running inside it. When working with offheap, the jvm will ask the OS, how much off heap can I use?. If the pod is configured without memory limit. It will tell the JVM that 26G is available.
That 26G will not be reserved
because others pods will also think they can use that.
so, having a pod with a hard 8G limit, will garantee that the JVM will not go over the fence.
m
ahh ok, I am actually using AWS node with 122GB of RAM and 26 GB is mem request
26GB mem request for single Pinot server
d
Ok!
then you can even bump it to 12GB xmx if the server pod has 26Gi request
m
Thanks @Daniel Lavoie, I increased the xmx from 4G to 8G and servers are up
I can use 12G
d
the rule of thump is ~50%
m
ok, How abt -Xms12G -Xmx12G
d
matching Xms is always a good thing in a container environment. that memory is wasted anyway if it is request by the pod
m
great Thanks for the help here
All the segments are in bad status and search not working. So I have to restore from backup. If the issue was OOM then I assume it might have caused some rouge segments. Now finding those and changing the offset in zookeeper to skip those will be hard..!!!
d
Are segments being reloaded?
They will remains in bad state until everything is reloaded.
m
Ohh triggered a reload now , see how it goes
d
watch your CPU and Disk IO, that’s a good tell of what’s happening
m
yes both spiked, especially disk
and now all segments came back to good and Pinot is trying to catchup with stream.
catching up very slowly though
d
cries in 5 hour segments reloads
👍 1
Open in Slack
PreviousNext
Powered by