Hi, so I'm trying to run pinot on a kubernetes cluster and to ingest realtime data from kafka. However, when I try to ingest data pinot seems to fail and stop ingesting data at a specific point. I tested this by running two different pinot kubernetes clusters at the same and having both ingest from kafka at the same time, yet they both also seemed to stop at around exactly the same time. If I run pinot on an individual machine it seems to work but for some reason it does not on kubernetes. The config Im running for pinot is 3 controllers, 30 servers, 1 minion, and 3 zookeepers.

its probably because of some issue with time column, Pinot tries to reason about the time column data when its creating a segment on disk.. If you check the server logs, you should see some exception

Druid might be a better solution for your use case. It's much more natively integrated with Kafka for ingestion

@Ethan Yu Do you see an errors in the debug endpoint (swagger)?

I was able to figure out the issue, apparently the controller was reaching its storage limit so we increased its volume size and now it works

Thanks for confirming @Ethan Yu