I am aware of the recommendation that segment sizes should be ~200MB and number of segments in a table should again be a few thousands as both can have a negative impact. I have a use case where in a REALTIME table, one column is blob data(other 2 columns will have strings with inverted index only) and its size is avg ~20KB. This leads to a situation where I either have too many small segments or moderate number of large sized segments(2GB). Going the route of too many segments is out of question as it affects zookeeper but is there a way I can support large sized segments? Based on this commit, it looks like large segment sized can be created(>2GB), is there any suggestion how to handle large segment sized table?

Are you making those string columns “noDictionary” ?

Yes, they are noDictionary

You can increase segment size to perhaps ~500MB in realtime table. And if you really want to increase more, may be use realtime to offline job

I believe the inverted index needs dictionary encoding?

yes. use varLengthDictionary for them

This table will have an offline table as well. Same situation there as well. Segment size vs segment count

Offline can have 2GB segments

~200GB per day, 30 days of retention = ~6TB

200GB / day of pinot segment? If not, whats the 200GB format

200GB is reported by Pinot table in testing.

that includes replication, btw

Yes, 2 replication

so 100GB for one replica (use this to get segment count)

Yes, ok

I am aware of several prod deployments which are using > 1 GB segment size. I’d say start with that.

Ok, in this case, this should affect the server host and the query performance and not have an impact on the controller? We have peer download for segments.

I don’t think it necessarily impacts query perf.

Ok, thank you Mayank for the quick help and guidance. Let me try out things as suggested.