Any idea why Pinot server JVM usage grows as records grows i

Question

Any idea why Pinot server JVM usage grows as records grows it does glow slowly However this way it will eventually have high heap usage I do have 5 6 inverted and sorted indexes in my table with very low cardinality columns using off heap and MMAP for segments in my setup have ~ 100 150 segments 500 600M rows gt continue to grow over 150M per day 6 server nodes 8GB is given to JVM and rest is available to off heap What I expected it will grow and as segments go to disk it should come back the lower bound I expected that this cycle should continue and the the lower bound of memory should remain constants Am i missing some setting

Mayank · Answer

Which version of Pinot Also are you see this increase only for server or broker controller as well If all perhaps related to Prometheus

Harish Bohara · Answer

0 10 0

Harish Bohara · Answer

Should I remove Prometheus and try

Mayank · Answer

First check if all components

Harish Bohara · Answer

Controller and broker looks ok to me i see controller and broker come to ~ lower bounds after GC then go up and again go the ~same value

Mayank · Answer

But you are just ingesting data so broker should not have any spikes at all

Mayank · Answer

It should be flat lined

Mayank · Answer

May be try without Prometheus just to debug

Harish Bohara · Answer

Sure

Harish Bohara · Answer

Got the issue Initially I had 2 tables both table have sane dat source this setup was giving all those issue with ZK disconnect very high GC time etc 1 Append table + with star tree index 2 Upsert table 1 2 bloom filter index I created 2 diff cluster 1 Append table without star tree large green circle a This grows slowly as i mentioned above however I ran full gc once amd it came down to 200MB which looks good b This means it does not have any memory lead and eventually a full gc will happen and will free up the memory 2 Upsert table 1 2 bloom filter index red circle a This setup grows very fast and ~200M records it goes to 8GB heap size b Then GC does not help event force GC does not recover memory c Out of book table configs need to debug what is wrong here Will debut issue in Upsert tables I will run a separate cluster with Append and Upset to see how it performs

Mayank · Answer

For 2 is your pk monotonically increasing They are stored on heap today Also have you noticed a need for bloom filter It is also stored on heap iirc it might be better to just use inv index which is off heap and will give you good perf

Harish Bohara · Answer

I though bloom filter will allow Pinot to skip a segment file few DB like C does this Ok I will remove it from bloom filter

Harish Bohara · Answer

For 2 my PK is a UUID

Harish Bohara · Answer

The setup 1 is working fine for 1 weeks with 2 tables each having 1B records I see the pinot server memory map usage is increasing to 30+ GB Is it normal I did not understand why it should grow It should be a flat line Am I missing something

Mayank · Answer

If data is being ingested then memory map usage is expected to increase with data growth