What is the optimal segment size for pinot tables? We are planning to merge parquet files daily/monthly and are researching for optimal file size. From what I have found, 100 MB (or 128 MB) is the sweet spot.

For real-time segments generated on servers anywhere 100MB - 300 MB. For offline it can go much higher

Thanks for the info. Is there any recommended limit on how big the segment can be? With bigger segment, I assume 1. queries will be faster since the broker doesn’t have to aggregate results from a lot of segments. 2. less load on zookeeper since the number of segments decreases and hence the segment metadata stored in zookeeper 3. the network latency between deepstore (S3) and local disk storage decreases since only a few segments have to be downloaded. I am thinking about creating segments with size around 1GB then.

When using large segments if you run group by queries which return thousands of groups then there is a configurable max group count limit per segment file. If you exceed this limit then you will not get all the results for all groups.

BTW, Broker doesn't do the segment level merge, server does. And on the other hand, more segments means parallelism, so there's a balance point. For offline, I have seen segment size can easily go 1GB (and ocassionally even > 1 GB)