Apache Pinot

:wave: I wanted to raise a topic that I think is worth discussing as Pinot adoption grows: ZooKeeper scaling under very high segment counts.

We’re running a deployment with some tables pushing past 500,000 segments. At that scale we’re starting to see real pressure on ZooKeeper. IdealState update times getting into the tens of seconds, periodic tasks that do full O(N) ZNode reads on every cycle, and the general concern that ZK isn’t really designed to be a data store at this scale (the commonly cited guidance is to keep ZNode counts well under 100k). Some simple API endpoints like listing all segments in the table start to breakdown as well.
Even approaches like segment rollups or logical tables help operationally but don’t fundamentally reduce ZNode count when you’re in the multi-hundreds-of-thousands range. Our infra teams are already a bit scared at the znode count and jute max buffer size we are approaching :worried:

Have some more thoughts in thread :thread:

I’ve been digging into the codebase and thinking about what a longer-term path might look like. Specifically, whether the segment metadata and assignment data (IdealState) could move to an embedded *Raft+RocksDB* store on the controllers, keeping ZooKeeper for what it’s good at: leader election, liveness, small coordination signals. Systems like Doris/StarRocks and Kafka (via KRaft) have taken this approach for similar reasons.

I was able to PoC the Raft+RocksDB approach on one of our clusters with success, so this has given me more motivation to look into this path.

A few questions:
1. Are there existing issues or design docs around this? I’d love to contribute to something already in flight rather than duplicate work.
2. If nothing exists, would the maintainers be open to a design doc in this space? Happy to write one up and share it for feedback before going anywhere near a PR.
3. Is anyone else running at 500k+ segments per table? Curious whether others are hitting the same ceiling, and if so whether you’re just running a beefier ZooKeeper cluster or have found other mitigations.
I’m just wondering if there’s appetite to tackle this more fundamentally, and whether there’s a place to collaborate on it. Thanks in advance!

We’ve gotten to 200-300k segments. That’s kind of the limit where yes, ZK operations get bad. We haven’t done this yet, but <https://docs.pinot.apache.org/architecture-and-concepts/components/table/logical-table|docs.pinot.apache.org/architecture-and-concepts/components/table/logical-table> is probably what solves this. You need to make separate physical tables

Yes we are considering logical tables as well, which would definitely help the big IdealState node updates, and other zk operations like list children etc.! And we may go down that path. I do worry this avoids the general high ZNode count issue though :worried:

one caveat with this is that all the underlying tables/shards need to have their schemas in sync. how are you managing this? its not really possible to manually do it for groups of 10s of physical tables

I imagine at this scale you're not doing things manually anymore 