I am running pinot in my local machine and have been doing some POC .
I was able to ingest 45 milion records from kafka .Now after this point I am getting a heap issue out of memory while I try to insert more data .
I am not able to understand from which side the issue is bc when I tried incresing heap on the controller last time I had this issue it worked for me and the messages started ingesting again.
is this an issue in the pinot or what am I doing wrong. Is the kafka topic full ?
m
Mayank
03/28/2025, 1:51 PM
Need more details, how are you running Pinot (QuickStart?), which component is OOM’ing, what’s the jvm config?
a
Aditya Verma
04/03/2025, 7:24 PM
Hi @Mayank
Its the controller Part as whenever I give it extra heap it starts working fine and the kafka messages starts getting inserted in the pinot .
I am guessing its the messages that are being held up in the memory .
please help me understand it.
When this happens the UI gets stuck and nothing is being loaded or ingested
m
Mayank
04/04/2025, 4:07 PM
That likely means you are running out of resources and your provisioning doesn't match your workload setup
a
Aditya Verma
04/05/2025, 3:59 PM
got it
Aditya Verma
04/05/2025, 4:00 PM
if I want to deploy pinot on kubernetes
what are the minimum things I should keep in mind
in term of deployment.
also what kind of storage should one buy ?