Hello team, we are still in POC phase with Apache Pinot for our use case, but we are happy go to go fully test the use case on production and i want to ask few questions in regards to capacity planning, i have gone through https://startree.ai/resources/capacity-planning-in-apache-pinot-part-1 few times, so here is my use case and questions, will be really grateful if you can assist: Use case: Data retention - infinite, all data is in hot tear Current data size - 12B rows with nesting/arrays (around 48B rows unnested) - each nested row is around 3kb(this is raw, without compression from pinot) of data when serialized in JSON bytes and 60ish columns in pinot. QPS requirements - 1500-2000 for current needs, but can be more in the future; P99 under 500ms, P90/50 under 100ms Kafka Ingestion Rate - 1000 msg/s high rate; during spikes we would need 20k-30k msg/s this is overall for a given topic (we currently have 100 partitions for the topic, but can scale even more if needed). In terms of per partition rates 10msg/s for normal and 300msg/s for spikes Daily ingestion rows - 12M per day with nesting/arrays Replication factor - 3 Segment size - 500MB (around 70k segments when all data is up * 3 (replication) = 210k segments) Types of queries - 99% of the queries are select Col1, Count, avg, sum,min,max from tbl where customerId = {customerId} groupBy col1 or similar queries, there are some use cases where we need to fetch raw data of up to 1000-2000 rows, but those are rare and not under the QPS requirements described above Number of tables - 1, possibly few more if we add DIM tables for join queries with nomenclatures, but currently we don't have/need it Questions: 1. Is it better to have many smaller machines or fewer bigger ones, as there are examples where you go up to 32 cores and 126GB of memory per machine 2. Is there a benefit to using offline and realtime tables in combination if all data is ingested through kafka and i see that our ingestion rate is not as high as in other use cases. As far as i have researched we should be fine using only realtime table without retention 3. Do you have any recommendations for the cluster: a. Controller/Zookeeper - 3x (16vCpu, 64GB Mem, 200GB storage each) - how do i calculate storage for controller/zookeeper (as far as i understood Controller and Zookeeper are singleton instances so multiplying them is just for redundancy) b. Broker - 5x(8vCpu, 32GB Mem) or 10x(4vCPU, 16GB Mem) ? c. Server - 20x(8vCpu, 64GB Mem, 2TB storage) or should i go with smaller/bigger machines

Thanks for providing the details: 1. It is a function of total cores and total memory, and number of nodes is rarely something I'd personally optimize for. But yeah, 8/16 cores with 64GB is most common for servers. 2. If your data flows through Kafka, you can start with realtime. You can always use techniques to decouple consuming vs serving nodes if at some point you want to do that. 3. Cluster: a. Controller/ZK size seem fine, you might also go 8 core 64GB + 200GB storage. Controller is not singleton per say, the periodic tasks are distributed. b. Start with 8 core 64 GB - 3 brokers c. Seems fine. I'd pick a number divisible by replication so that there is no data skew. For perf: • Use replica groups • Pick the right sorted index and partitioning to avoid scanning data • Timestamp index for time. Store time in the finest granularity you care for querying (eg seconds/hours instead of millis). If you are asking about using deepstore this is the right doc, not the one you shared. If schema is more or less fixed and it is mostly about unnesting, store unnested data. Number of rows increasing is not a big issue for perf.

apiVersion: v1
kind: ConfigMap
metadata:
  name: pinot-controller-config
  labels:
    app: pinot
    component: controller
data:
  pinot-controller.conf: |-
    # Controller host and port
    controller.host=pinot-controller
    controller.port=9000
    pinot.set.instance.id.to.hostname=true
    # Cluster name
    controller.helix.cluster.name=pinot-quickstart
    # ZooKeeper configuration
    controller.zk.str=pinot-zookeeper:2181
    # External access
    controller.access.protocols.http.port=9000
    # Controller periodic tasks
    controller.task.scheduler.enabled=true
    controller.task.frequencyPeriod=5m
    # System resources
    controller.data.dir=/var/pinot/controller/data
    controller.local.temp.dir=/tmp/pinot-controller-tmp
    controller.deleted.segments.retentionInDays=1
    # Optional: Throttling config for high volume
    controller.realtime.segment.commit.timeoutMs=120000
    # tuning
    controller.allow.hlc.tables=false
    controller.enable.split.commit=true

apiVersion: v1
kind: ConfigMap
metadata:
  name: pinot-broker-config
  labels:
    app: pinot
    component: broker
data:
  pinot-broker.conf: |-
    pinot.broker.client.queryPort=8099
    pinot.broker.routing.table.builder.class=random
    pinot.set.instance.id.to.hostname=true
    pinot.query.server.port=7321
    pinot.query.runner.port=7732
    pinot.broker.adaptive.server.selector.enable.stats.collection=true
    pinot.broker.adaptive.server.selector.type=HYBRID
    pinot.broker.netty.native.transports.enabled=true

apiVersion: v1
kind: ConfigMap
metadata:
  name: pinot-server-config
  labels:
    app: pinot
    component: server
data:
  pinot-server.conf: |-
    pinot.server.netty.port=8098
    pinot.server.adminapi.port=8097
    pinot.server.instance.dataDir=/var/pinot/server/data/index
    pinot.server.instance.segmentTarDir=/var/pinot/server/data/segment
    pinot.set.instance.id.to.hostname=true
    pinot.server.instance.realtime.alloc.offheap=true
    pinot.query.server.port=7321
    pinot.query.runner.port=7732
    realtime.segment.serverUploadToDeepStore = true
    pinot.server.instance.segment.stream.download.untar=true
    pinot.server.instance.segment.stream.download.untar.rate.limit.bytes.per.sec=100000000
    pinot.server.instance.table.level.max.parallel.segment.downloads=4
    pinot.server.instance.enable.split.commit=true
    pinot.server.netty.native.transports.enabled=true

1. With the above config, how would deep store work, because i don't think i understand it completely. From what i have seen/tested, if no deep store factory class is setup in controller/server(realtime.segment.serverUploadToDeepStore=true) then the "deepstore" is the actual volume that is attached to the k8s pod that runs with it? 2. The above config was used during Testing on DEV and with each performance tune(based on the docs/github issues) i have seen some improvements, do you have any other suggestions that i can use in terms of infrastructure setup? 3. Do you have recommended version of zookeeper to run or whatever is fine? We run 3.5.5, which is quite old, but want to ask if there is some corellation PS: i have updated the cluster as you suggested. As for indexes and timestamps we have this covered already, we have not yet used replica groups as on dev we had around 8-9 times smaller infra and only 25m rows, while still achieving over 1200 QPS, but will need to dig into it.