Hello, I have a Pinot deployment done through K8s wherein I'm progressively adding fields to a realtime table. This deployment is very basic (1 server instance only, with 6GB for java Heap, the pod has a memory limit of 7GB, 100GB persistent storage, deep storage for segments has been enabled) but I'm getting multiple server restarts because the pod keeps getting killed with OutOfMemory errors while ingesting data and creating segments. It seems the cause is not the JVM itself but off-heap memory maps, please see the following image for more details.

Server naturally uses off-heap

Thank you for the feedback Daniel. What metrics in particular should I take a look at?

standard container and jvm metrics

From the grafana chart I put above it seems Pinot is using >2x the memory for offheap than JVM rather than ~100%. With such a high amount of memory mapped usage does it make sense to add more servers with the 50% for offheap rule?

This is not the metric I meant

Ok, so if I understand correctly. The processes running in my server pod are trying to use more memory than the pod has available (limit: 7GB)

Exactly

The JVM usage is as follows:

Just reduce the heap

Does that sound about right Daniel?

Yes

Isn't there a concern that such a small heap won't be enough to hold the segments in memory for fast querying?

offheap is fast

Thank you for the assistance, I will try out this config.

Having more servers means you reduce the impact of redistributing or redownloading segments

What is the event rate from your event stream, and how many partitions do you have

16 partitions. It is a scheduled cron, outputting daily ~50M entries. We are in the process of removing the cron job to an event-based stream.

Parts or consuming segment are on direct memory that can OOM if not enough (unlike mmap).

That will cause a JVM OOM, not a K8S OOM

Oh sorry, long thread, I assumed it was JVM OOM.

How can I distinguish them, in K8s? Via the pod logs?

Yes,

Could we confirm if it is JVM or pod OOM? Also if it is JVM, is it heap or direct memory OOM?

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  [thread 140238581155584 also had an error]
SIGBUS (0x7) at pc=0x00007f8c70a013c2, pid=9, tid=0x00007f8bd167c700
#
# JRE version: OpenJDK Runtime Environment (8.0_292-b10) (build 1.8.0_292-b10)
# Java VM: OpenJDK 64-Bit Server VM (25.292-b10 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# v  ~StubRoutines::jbyte_disjoint_arraycopy
#
# Core dump written. Default location: /opt/pinot/core or core.9
#
[thread 140238499997440 also had an error]
# An error report file with more information is saved as:
# /opt/pinot/hs_err_pid9.log
[thread 140237039417088 also had an error]
[thread 140238498944768 also had an error]
[thread 140238586418944 also had an error]
[thread 140238396491520 also had an error]
[thread 140238582208256 also had an error]
[thread 140238587471616 also had an error]
#
# If you would like to submit a bug report, please visit:
#   <http://bugreport.java.com/bugreport/crash.jsp>
#
Aborted (core dumped)

Got this with a 3GB java heap and a pod memory request of 6GB (limit 7GB)

Oh!

134 error code, yh, no logs though

What does the hs_err.log say?

Do you mean reducing the number of partitions in kafka?

Kafka. But let’s first find out if heap or direct memory

I'm trying to access the log file but the particular fs path where it is stored is not in a pvc

Anything in server log?

Exception in thread "HitExecutionView__12__49__20210524T1141Z" java.lang.InternalError: a fault occurred in a recent unsafe memory access operation in compiled Java code

Off-heap memory I assume?

Are you using any groovy functions?

Yes

What does it do

Parses a time string in a wierd format from a json field, to the java standard and returns the milliseconds since epoch of that time.

Ok. My guess is the heap OOM

That is the last line in log from the server.

Yeah but can you check if there was segment generation happening around that time

I can't see anything referring to segment generation: https://pastebin.com/D8bxdZeA

I think the relevent logs are only available inside the pod (stdout only shows WARN). The INFO file is lost on restart with the default configs.

If you want to optimize cost then you can throttle the burst so consumption event rate is lower, reduce num parritojis in Kafka once max event rate is low, and limit number of parallel segment generations

I see some INFO level logs in stdout

Or else just add more vms 😀

At this point, simply having a formula that lets me know the memory requirements and size up the servers, would be good enough

Yeah there is a real-time provision tool in the docs

https://github.com/apache/incubator-pinot/blob/master/pinot-tools/src/main/java/or[…]inot/tools/admin/command/RealtimeProvisioningHelperCommand.java ?

Yes

I'll take a look, get back to you soon. Thank you both so much for the help

I think you should find it within the pinot image as a standalone script in

bin

Is the tool meant to take a long time? Been running for 5m.

docker run --rm -v /home/pedro/dev/Pinot:/tmp/volume apachepinot/pinot:release-0.7.1 RealtimeProvisioningHelper \
-ingestionRate 1000 \
-numPartitions 16 \
-retentionHours 720 \
-numRows 50000000 \
-tableConfigFile /tmp/volume/specs/tables/HitExecutionView_REALTIME.json \
-schemaWithMetadataFile /tmp/volume/specs/schemas/HitExecutionView.json
Executing command: RealtimeProvisioningHelper -tableConfigFile /tmp/volume/specs/tables/HitExecutionView_REALTIME.json -numPartitions 16 -pushFrequency null -numHosts 2,4,6,8,10,12,14,16 -numHours 2,3,4,5,6,7,8,9,10,11,12 -schemaWithMetadataFile /tmp/volume/specs/schemas/HitExecutionView.json -numRows 50000000 -ingestionRate 1000 -maxUsableHostMemory 48G -retentionHours 720