Hi everyone, We are running a large-scale Pinot deployment and plan to store all our data in a single, large table. As our segment count grows into the hundreds of thousands, we are already hitting the ZooKeeper ZNode size limit (

jute.maxbuffer

) due to the large segment metadata. We have reviewed the official troubleshooting documentation, which suggests two primary solutions: 1. Decrease the number of segments: We cannot use rollups or further merge segments, as our current segment size is already optimized at ~300MB, and we need to maintain data granularity for our query performance. 2. Increase `jute.maxbuffer`: We view this as a last resort, as we are concerned about the potential downstream performance impacts on the ZooKeeper cluster. Given these constraints, we have a few questions: • What are the recommended strategies for managing ZNode size in a table with a very high segment count, beyond the two options mentioned above? • Is there a practical or theoretical upper limit on the number of segments a single Pinot table can efficiently handle before ZK performance degrades? • Are there alternative configurations or architectural approaches we should consider for this scenario?

You can always merge smaller 300 MB segments into larger ones. Can go up to 1GB or even more. What’s the jutebuffer size you have now, we have seen it increased regularly in large deployments