Pinot server: single big LV vs multiple mount points for segment storage?

Pinot does parallel reads, yes. So you should optimize for what gives you better read IOPS and throughput

Could you confirm if Pinot supports writing data across multiple disks, or if it requires configuration changes to handle that setup?

Unless you are looking for tiering, you should use a single data Dir.

Ok, I’ll try tiering and start with streaming from Kafka first. While creating the table, I’ll define both hot and cold directories. I still need to explore how to configure this properly, since I have 12 disks and would like to set up retention of data on each disk.

What is the use case you are trying to solve? Why do you have 12 disks attached to a single server (typically this is not how people deploy Pinot)?

We already have this hardware available. Our use case is that we’re ingesting around 15K TPS from Kafka (JSON format) into Pinot. The data volume is huge — we need to retain about 6 months of data (~40TB). Our cluster has 8 nodes, each with 128GB RAM and multiple CPU cores, and each node comes with 12 local disks. That’s why we’re looking at how best to configure Pinot storage across these disks for both performance and retention.

Suggestion: use a single dataDir on top of mdadm RAID0 (striped) across the local disks. Pinot parallelizes reads at the segment level; a striped FS will give you the best aggregate read IOPS/throughput and keeps ops simple vs juggling many mount points. Why RAID0 here • Pinot’s durability comes from replication across servers + deep store, not single-disk redundancy. With RF≥2, a node loss is tolerable; RAID0 maximizes performance per box. • Multiple dataDir paths work but tend to fill unevenly and complicate balancing/recovery. One big FS avoids hot-spotting and path management. • LVM concat/linear gives capacity but *no parallelism*; mdadm RAID0 stripes IO across all spindles. Gotchas / answers • Parallelism: Pinot already does parallel reads; striped FS ⇒ more devices servicing those reads. • Many paths in pinot.server.instance.dataDir: supported, just harder to operate (capacity fragmentation, rebalancing quirks). • Perf & recovery: RAID0 generally faster for segment fetch/serve. If a disk dies, the node is toast (as with any JBOD’d dataDir on that disk), but the cluster will *rebalance*; set RF≥2 (ideally 3) and use deep store (e.g., S3) for quick reloads. • If you need local redundancy: pick RAID10, but expect ~50% capacity hit.

With RF=2 and RAID0, ingest capacity should be calculated at half the nodes assuming a catastrophic loss of half the nodes in the cluster (over-simplifying). The rebuild will also consume existing capacity and will impact ingest imo,

To rebuild the node if comes up with empty, Pinot will be mainly download segments from deep store and only for the current consuming segments from Kafka

Interesting

nodes for data serving