Hi Team, I am facing a

java.lang.OutOfMemoryError: Java heap space

issue while running a realtime upsert table in Pinot. Setup details: • Table type:

REALTIME

(upsert = FULL mode) • Primary keys: ~2B (hashFunction = MURMUR3) • Cluster: 4 servers • Memory: 128 GB per server Problem: Whenever I start the servers, memory usage grows continuously until ~100–110 GB, and then the server eventually hits:

java.lang.OutOfMemoryError: Java heap space

attaching heap memory use trend, server config, table config, server logs and gc logs

In OSS implementation, upsert metadata is kept on heap. From our experience, one sever with 32GB can keep 100M PKs in heap. And typically, you don’t want to go beyond 32 GB heap so that means you need to scale horizontally

Considering that each PK uses around 150 bytes (including overhead), for 100 million PKs the total heap memory utilization should be approximately 14 GB. The remaining 15–18 GB should be free, but the heap usage still increases to around 31 GB before garbage collection (GC) runs, after which it drops to about 20 GB. Why is this happening — could there be a memory leak?

You can take a heap dump and figure out, but my guess is that it is mostly attributed to PKs and mutable index (inverted and range).

The heap dump is quite large around 35 GB and none of the available free tools are able to analyze the heap dump. We also observed that when we start a server instance with 32 GB of memory but do not begin ingestion through this server (no PK is present), the memory utilization gradually increases to around 20 GB. After that, the garbage collector (GC) runs and reduces it to approximately 2–3 GB.

You can use jstack