I know its after a long time, But just wanted to update one thing . > On this specific Pinot server, all tables are upsert-enabled with a combined disk utilization of 612GB. What could be causing the heap to spike to 700GB? During Pinot server startup, the heap fills up rapidly, leading to continuous Full GC pauses and OOM failures. Upon debugging, we got to know that the

validDocIds

snapshot was missing from many segments. Assumption 1. Dataset: We are running a 10 TB Upsert table containing millions of rows with a high-cardinality primary key. 2. Restart Behavior: When a server hosting an Upsert table restarts, it must rebuild the in-memory Upsert Metadata Manager (the hash map tracking

PrimaryKey -> RecordLocation

). 3. Segments with Snapshots: The server can efficiently skip entire segments if their maximum timestamp falls outside the metadata TTL, keeping heap usage low. 4. Segments without Snapshots: The server is forced to load the entire segment into the heap to determine which records are valid. A background TTL filter job (running , lets say every 10 minutes) is responsible for cleaning up expired keys. Now i feel like the 4th point is causing this issue of high heap , If yes ( 4th point is correct ) Is there any way we can create snapshot for segments thats are missing them..? If no ( 4th point is wrong ) Can you please let me know, what other reason can be there ..? @Xiang Fu, @Mayank

How many PKs do you have in that server? Does the heap go down after metdata is built? If yes, then it is just temporary garbage that can be GC'd. If not, then likely it is needed to hold the upsert metadata.

I feel like, GC is not able to clean up , in same phase server is loading into the memory . We had around 125M Primary Keys, and 8 hours of MetadataTTL ,so all these are not actively needed to loaded onto heap . But due to the missing snapshot , It has to go with the full scan ( i believe ) And no , Heap won't go down, It will hit full GC, and the server will be halted . And the restart was going through on the severs that snapshot are available for almost all the segments

Then I feel it is the metadata that is occupying the heap. You can take heap dump or jmap to confirm

what's your servers memory size?

We have BM, which has 1TB memory,

what's the jvm setting

do you know the in memory size of the primary key table? one quick thing to try is double your xmx and see if gc pressure got any relief

We did since the startup was failing, We did with 650 GB Heap as well, Which also caused Full GC, Now the Pinot servers are stable. Just an overview of the issue, One more time 1. We faced an issue, which required a restart of Pinot Server ( Before restart, Heap usage was around 70 GB) and xms, xmx set to 120 GB 2. While we did the restart , We observed that, Heap is getting full, and hitting full GC in some servers ( Not on all of them ) 3. We tried increasing Heap config ( 120 GB -> 220 GB -> 450 GB -> 650 GB ) all of which was failing Now if you look at first point , server was healthy during running phase , But when restarted, Heap issue started happening , due to maybe below point https://apache-pinot.slack.com/archives/C011C9JHN7R/p1779982687874809?thread_ts=1767894697.796559&cid=C011C9JHN7R

hmmm, do you have the estimate on the Primary key size

We use composite key (4 columns, which all are string type), ~100 - 200 bytes per PK

650GB is wayyyy too much. You probably want to scale more horizontally (add more smaller servers) than vertically

Yes, agree, But the issue is why only on the server restart ( Maybe be due to snapshots were missing, Then why snapshots were missing.? ) Before restart it was only using around 70GB heap.

Hard to debug over slack. For debugging, you'd want to take heap dump and/or jmap to see live objects vs garbage. By construction, if upsert metadata takes X amount of heap in steady state, it should also take the same amount during restart in terms of non garbage collectable memory. Rest should be garbage - as in just because of restart it should not need >> X amount, regardless of snapshot or other factors.

Rest is garbage makes perfect sense . And its not upsert Metadata which is causing the heap, but it is the garbage , which is a byproduct of a full scan on the snapshot missing segments . Can this be possible ..? And as you mentioned, We will take a heap dump and see the live vs garbage object ratio