Hi everyone, I’d like to ask a question. My Pinot server has a high specification—96 cores and 1TB of memory. But I’m wondering: even with powerful hardware, is there still an ideal upper limit to the number of segments a single server should handle? Would it be more advantageous to use multiple medium-spec servers instead of a few high-spec ones? Currently, one of my servers is handling around 130,000 segments, and I often run into issues where Prometheus fails to scrape metrics with `connection reset by peer errors`*.*

Hi Wilson. 130k segments should be fine memory-wise, though metadata could take a good chunk of heap, especially if tables are wide. 1.4 release should be a bit better in this regard (e.g. https://github.com/apache/pinot/pull/15591). I think the issue you mentioned is caused by: 1. really slow and GC-unfriendly code in prometheus jmx exporter 2. large number of segments so generating metrics takes time and increases GC pressure. You might reduce the number of errors by increasing connection/response timeouts or altering rules in yml file used to configure exporter. I'd recommend capturing a profile with async-profiler and opening issue in GitHub.

What’s the jvm configs you have? Is the node also running other pods/containers?

Hi @Mayank, JVM configuration is also something I’m looking into for potential tuning and optimization. Previously, our settings were:

-XX:ActiveProcessorCount=86 -Xms16G -Xmx728G -XX:+UseG1GC
-XX:MaxDirectMemorySize=228G

Later, we ran into issues related to mmap memory, so we changed the configuration to:

-XX:ActiveProcessorCount=86 -Xms196G -Xmx196G
-XX:MaxDirectMemorySize=256G

Yeah most deployments run Pinot multiple medium instances (say 8 or 16 cores, 64 or 128 GB), so you can scale horizontally. Any reason you went the other way?

The original idea was to avoid running multiple Pinot server pods on the same Kubernetes node, and to give each server more resources for tuning. But now I’m trying to evaluate the following: 1. Would it be better to run multiple medium-sized Pinot server pods on this node (96 cores, 1TB memory) to increase the overall number of servers? 2. For a medium-sized Pinot server, what would be a reasonable range for the number of segments it should handle?

Hmm, do you not have smaller nodes where you can run one Pinot server pod each?

If the recommendation is to run only one Pinot server pod per node, then we would consider purchasing smaller nodes dedicated specifically for Pinot servers.

Yes, that is how most deployments have it.