I'm looking for best practices for choosing hardwa...
# general
c
I'm looking for best practices for choosing hardware (cpu, memory, storage). This is the only documentation I found. https://docs.pinot.apache.org/basics/getting-started/frequent-questions/operations-faq#how-much-heap-should-i-allocate-for-my-pinot-instances Of course I know it depends on my workload, but can someone give me a general hint or experience? Anything is fine.
m
Like you said, it depends a lot on workload. Instead of giving a very generic answer that might mislead you, I’d say setup a single broker + server cluster and see how much you can squeeze out of it, and then simply scale up/out.
c
Thanks Mayank, I know your opinion is the best approach. However, if there is a best practice for selecting hardware for each type of workload, I will be able to choose hardware for my workload. Am I unable to find these best practices?
m
Servers do most of the work, so you want to use higher cpu/mem/storage for server nodes. If you load 1 TB of data on each node, ensure you have SSD (ebs), perhaps at least 16 core/64GB memory. Don’t allocate too much heap space. If 64GB ram, we do 16GB xms=xmx.
Brokers/Controllers don’t need local attached storage. Always use deepstore (or NAS/NFS if not on cloud).
cc @User
c
Mayank, thank you as always. "Don't allocate too much heap space. If 64GB ram, we do 16GB xms=xmx" Question) Will the remaining memory be used by MMAP? Question) If we can choose 128G of memory, can we expect higher performance than 64G? Or, even if you choose a memory larger than 64G, is it difficult to expect a proportional performance improvement?