Hi Folks I am facing issues due to spikes in laten...
# troubleshooting
n
Hi Folks I am facing issues due to spikes in latency in Pinot. I have observed that pinot server’s memory is fluctuating between these specific values. At same time latency also decreases and increases with same pattern. Just wanted to know is it due to heap memory or GC? Or what values should i keep? Current specifications -: Number of severs - 3 Heap - 5 GB Memory min max - 8 GB and 10 GB CPU min max - 4 and 6 I have setup pinot using helm chart given on official documentation - https://docs.pinot.apache.org/basics/getting-started/kubernetes-quickstart Attaching screenshot for latency and memory
m
Is this for realtime or offline? If realtime, another possibility is periodic segment commit. Is this causing a latncy issue?
n
I tried it on realtime. I observed a latency issue as latency is moving between a range from 10ms to 25ms and at same time i am observing a pattern between memory. Memory also keeps on moving between a set boundry for broker and server
Memory is moving in a range and making a pattern like this
At some point it is getting choked then it goes till lower level and then again goes till top, similar is the case with latency against it
m
If you give 5GB heap, this sawtooth seems reasonable to me.
n
Yes i have 5 GB Heap with 3 servers running in pods
What is the recommended heap size ?
m
Instead of debugging it from memory footprint side, I’d rather try to understand your workload and your SLA requirements.
That would be a better way to optimize your cluster.
n
We have approximately 12 GB Data divided into 170 segments over 3 servers, 3 brokers
We are expecting latency of 10 ms as constant but we are getting it in form of pattern from 10ms to 25ms.
We are running 3 nodes in kubernetes 8 core, 32 GB RAM, 100 GB SSD in GCP
m
But latency is function of query selectivity among other things. Are all queries guaranteed to be equally expensive? Is it possible some queries are more expensive (and also use more memory) than others?
n
query pattern is fixed and they are equally expensive and qps is also nearly constant.
pinot.server.instance.realtime.alloc.offheap=true This property was set in helm, can this property cause the issue in heap? And, can you please tell what is the role of this property
m
This will allocate consuming segments off-heap. The purpose is to avoid GC.