How large has a single Pinot table been scaled in production? Anyone running > 150TB in single prod table? Looking for real-world upper bounds on one table (not cluster-wide totals) - specifically total row count, on-disk size, and segment count. Questions: 1. What's the largest single table you run, by rows / bytes / segment count? 2. What became the bottleneck first as segment count grew — ZK metadata + ideal-state size, Helix state transitions, broker routing-table build, controller rebalance duration, server startup/mmap time, or minion throughput? 3. Is there a practical segment-count ceiling per table (and per server) before ops pain starts? 4. At this scale, what segment size are you targeting? We're wondering whether fewer/larger segments is the right trade against query-time pruning. 5. Anyone running >100K segments in one table — what tuning was required to keep it healthy?

1. single replica: 500B/ 500 TB/ 500k 2. increase zk jute buffer size , you can vertically scale up zk; 3. we usually recommend running under <100k 4. There is a limit for single column <2G, the segment size can go larger like 1-4GB 5. really depends on your workload, realtime/batch/which minion tasks running , query pattern, etc. Usually recommend to configure replica group so operations will be simpler, e.g. you can do batch restart/upgrade etc.

Thanks @Xiang Fu, any known min and max safe limits for Djute.maxbuffer, what's the max we can configure for 300k segments, in future it can grow up to 500k segments ?

4MB - 8MB is actually safe to go with , Pinot Official doc is also suggesting the same , 500K segment can be handled with this config

@Xiang Fu / @Badhusha Muhammed whats the max number we can increase for Djute.maxbuffer for large clusters ? Have you increased it any prod system?

for jute buffer in prod we are running with 128mb or even larger, which requires bigger zk boxes, but still pretty stable

does this not put any extra load on serde ops? 128mb is quite large for a zk jute.buffer config

Yes, mainly for large ideal states/external view

github.com/apache/pinot/issues/19240 could this issue be somewhat linked? @Xiang Fu it looks like a state mismatch

would rebalance solve the issue? if they are not tenant, then technically consuming segments may not be created on those servers, then you won't see ingestion. Once consuming segment is sealed, pinot controller will create new consuming segment on the same server

yeah rebalance fixed it, but why didnt adding the tenant label to those servers create new consuming segments on them is my query

This is expected. I think when you create the table, there are fewer servers. Then partition mapping won't change after table creation.

the number of servers remained the same i removed 2 and added 2 🤔

When you created the table, how many servers in the tenant

i had 6 servers, with 6 consuming segments each (rf=2) i then replcaed 2 of them with 2 other servers that have the exact same hardware config. what happened was that while the newer servers had the tenant tag set to tenant X on them, the zookeeper dashboard did not show these as part of that tenant, rather it showed the older servers as still belonging to tenant X. in parallel, the older servers did not have the tenant X tag, but the zk thought they were a part of tenant X. have attached relevant images in the github issue

i think it's mostly the UI side issue, technically you cannot change server tag if the server still contains data from tables in that tenant

ah ok, thanks!