Hi Folks I am facing issues due to spikes in latency in Pinot. I have observed that pinot server’s memory is fluctuating between these specific values. At same time latency also decreases and increases with same pattern. Just wanted to know is it due to heap memory or GC? Or what values should i keep? Current specifications -: Number of severs - 3 Heap - 5 GB Memory min max - 8 GB and 10 GB CPU min max - 4 and 6 I have setup pinot using helm chart given on official documentation - https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart Attaching screenshot for latency and memory

Is this for realtime or offline? If realtime, another possibility is periodic segment commit. Is this causing a latncy issue?

I tried it on realtime. I observed a latency issue as latency is moving between a range from 10ms to 25ms and at same time i am observing a pattern between memory. Memory also keeps on moving between a set boundry for broker and server

If you give 5GB heap, this sawtooth seems reasonable to me.

Yes i have 5 GB Heap with 3 servers running in pods

Instead of debugging it from memory footprint side, I’d rather try to understand your workload and your SLA requirements.

We have approximately 12 GB Data divided into 170 segments over 3 servers, 3 brokers

But latency is function of query selectivity among other things. Are all queries guaranteed to be equally expensive? Is it possible some queries are more expensive (and also use more memory) than others?

query pattern is fixed and they are equally expensive and qps is also nearly constant.

This will allocate consuming segments off-heap. The purpose is to avoid GC.