Hi team, we have

"realtime.max.parallel.segment.builds"

that controls the number of parallel segment builds on realtime servers. I notice that the default value is 0 (which I think implies unconstrained, which is inherently capped by number of partitions). I am curious to know if folks have found this default to be good? If not, should we have a more conservative default (say 4)? cc @User @User @User

I am not sure what a good default is, since that is decided by the way the ingestion stream behaves, and the hardware that is available. I would leave it at zero and let installations tune it if they see a bottleneck in this area.

Yes, agree there's no one good default that works for all. However, what we are observing is that most folks who try Pinot for the first time and play with it, they quickly run into real time consumption issues. And a lot of time this traces back to OOM due to segment build. This leaves a bad impression for first time users (have seen this multiple times in the OSS now).

whats the implications of this? some segments will wait for sealing?

Max of 4 segments (in the default case) will be built on the each realtime server. The others will wait for a semaphore.

we used 2 at Uber in prod. this value will impact the realtime ingest and helix state change latency as far as I know.

If you are setting a non-default value, then there's no impact. For folks who don't are not setting a default value, and were somehow building > 4 segments in parallel, they may see only 4 segments being generated in parallel. Other segments will be waiting for semaphore. This is a protection against (heap OOM) that we have seen multiple times in OSS now.

Thanks for the explanation. I think 4 is a reasonable default