Yea, I've been having daily docker-for-desktop issues. I figured I was overloading it and weird issues were happening.

yeah, deskopt docker is only meant for simple quickstart use cases, docker for desktop hogs the cpu's for some reason. Heard mini-kube is better but haven't tried it

better to move to AWS or any other cloud instance for prod setup

Cool, thanks!

Yea, restarting docker-for-desktop fixed it. 🤷

Hi, we are doing a PT for 10k rps with 3 pinot servers each of 10 core, 10gb - 2 servers seem to be responding back in 1 ms , but one server keeps fluctuating upto 15 ms. All these numbers just for querying. I checked logs and garbage collection - logs says schedulerWaitMs for server that gave higher query time is obviously high.

Most of the times one server is on higher side

before we debug further, just want to let you know that you are already getting awesome perf numbers :)

at this point you are trying to optimize the 95th percentile latency

you can try few strategies • Partition aware routing • Replica group based placement strategy • Relocator strategy where you can move completed segments to other nodes

I am assuming partition aware routing is same as column wise partitioning from producer side. We added that in place, will do a quick test to validate. I searched for more details on configurations for replica group based strategy could not find how to do it..

And just to add we have 20 partitions and 2 replicas which means we have 40 consuming threads

its a bit advanced @Seunghyun Do we have any docs?

@Neha Pawar any docs on relocator configuration?

And we tried querying @ 10k rps and parallel ingestion at 2k rps, the response times at servers are 1ms, 1 ms and 17 ms.

anything special about that 3rd server 17 ms?

Ya just that the waitScheduleMs from logs is higher

Garbage collection is higher

Rest is exact replica of the other two servera

One of these 3 servers is behaving on higher side every time

is this for realtime table?

Yes realtime

realtime side replica group has been added by @Jackie recently

Relocator feature: https://docs.pinot.apache.org/operators/operating-pinot/tuning/realtime#moving-completed-segments-to-different-hosts

hmm i don’t think that we have a good documentation on how to use replica group 😞

for now, point at the design doc?

Ya, i checked the design doc..

we have answered this in a new FAQs page we're working on. it's still a draft

Can u pls give me the faq page pls

@Kenny Bastani can we publish the draft? we can mark it WIP keep working on it