I added two more servers to my cluster, and performance has dropped. One theory is that one or both of these new servers is slower than the previous servers, and thus causing the drop in performance. How can I confirm or refute that theory? Are there Pinot metrics I should be examining?

You can check server side latency metric. That will tell you the latency from individual servers, and identify if the new ones are slow.

Broker logs are good option, as I was hoping for something that wouldn’t require me to rig up metrics just yet.

If you scatter queries to N+2 servers instead of N, that may increase latency due to GC (assuming all servers are of same capacity). The probability that any one server is delayed due to GC is now higher. Of course, it depends on N. I don't think this may be observable if N is 10, but it may well be if N is 2.

N is 3 (went from 3 to 5). But seems unlikely to be GC related, as (a) it’s repeatable, and (b) time went from about 300ms to 900ms consistently.

Yeah, the broker log will tell you exactly which server took how long, and you can deduce broker side time (which will slightly increase due to more work for 'gather' phase).

Adding nodes to reduce latency

Not sure, perhaps there's a log4j setting?

You’re right - there’s an

immediateFlush="false"

flag in the pinot-broker-log4j2.xml file.

Oh nice, I actually didn't know.

And just FYI, the change in performance was due to a change in the star tree index that happened to be made at the same time, where net-net was we wound up with a lot more nodes in the tree.

#toomanymovingparts

🙂 Thanks again for the help.

I see. Glad you were able to find the root cause.