Hello all! Looking for some capacity planning advi...
# general
s
Hello all! Looking for some capacity planning advice based on a recent requirement change we will have 2 kafka topics writing about the same number of records per second each (b/w 4-6k per second) to two different realtime tables. Each table in pinot will end up having a 90 day retention and be about 7.5TB of storage (15TB w/ replication set to 2). So two pinot tables, about 15TB each, for a total of 30TB. Our read qps will be somewhat low, maybe 5 qps. We have control over the number of kafka partitions in each topic as well. So what I'm looking for is the optimum number of realtime replicas and storage/mem/cpu sizes. I have a few ideas, but curious what others might say. Thanks!
One note, we're going to use the tagOverrideConfig to move consuming segments to different servers. So our consuming servers will be small.
m
It also dependents on how or lo selectivity you queries are and the overall read workload. But generally speaking you can start with loading 1-2 TB per sever (assuming the sizes you mentioned are in Pinot segment size).