I have a question about using replica groups in Pinot. Replica groups help control query fanout, but is there any recommendation for how many servers a single tenant should have before it makes sense to enable replica groups? In my experience, when the number of servers is small, using replica groups actually makes queries slower because the load concentrates on fewer servers. Switching back to balanced segment assignment makes queries faster. I’m not sure at what server count (per tenant) it’s recommended to start using replica groups, and how I should properly observe or measure query fanout to decide when it becomes a problem. Any guidance or best practices on this would be appreciated.

we’ve seen the same thing. today, it’s just separate concerns. replica groups are designed to • isolate the infra for reliability • group the infra so it’s easier/faster to do rolling restarts I think you could do something like queries route balanced, but segment assignment is replica group based which should theoretically have Pinot automatically route to all available servers, but still be resilient to an entire replica group going down. But I haven’t tested this setup

you can still control the routing strategy - https://docs.pinot.apache.org/operate-pinot/tuning/routing

@Wilson and @Johan Adami I am surprised that you see slower queries when using a lower number of servers in combination with replica groups. Can you give an idea of what is a "low" number? (both number of replicas as well as replica group size). Are you using mirrored replica groups? Do you ue Adaptive Server Selection? Have you seen hot segments? Is there a wide variation in the number of rows in each segment? Pinot does segment assignment based on number of segements per server , and not based on segment size or number of rows in segments.

I wouldn’t say it happens always, it’s just something I’ve observed in the past. When number of segments queried >> number of servers, at lower query throughput you get better performance using all the servers since there’s just more CPUs to process in parallel. So when you move from balanced to replica group, you only get 1/num_replica_groups) for the same work. Though with high enough query throughput, it’s effectively the same performance.

Ah ok. I understand now, thanks for the explanation. On a low throughput yes, but then you are probably under-utilized on storage if you use large number of servers. That is a valid trade-off for low latency, yes.

In my current use case, I query Pinot through Trino. Recently, I’ve been seeing errors like

[BaseStreamingCombineOperator] Timed out while polling results block...

. These errors are timeouts that occur when Trino queries Pinot servers directly over gRPC. Right now, I have 6 servers divided into 2 replica groups. Under some large queries, the error above appears frequently. My understanding is that, for example, if a single table has 1,000 segments that all need to be scanned, then under a replica group–based strategy each server ends up processing about 1,000/3 segments. However, with a balanced segment assignment strategy, each server only needs to process about 1,000/6 segments. After switching to the balanced strategy, the load on each server was noticeably reduced and these errors became less frequent. That said, our Pinot servers are still being gradually added, and we plan to eventually scale up to around 20–30 servers. That’s why I’m trying to understand at what point (in terms of cluster size) it makes sense to switch over to using replica groups. I’d also like to ask if there are good ways to observe or detect hot segments.

I’m trying to understand at what point (in terms of cluster size) it makes sense to switch over to using replica groups.

it’s not about cluster size, it’s about the ops experience/reliability. replica groups give you isolated sets of servers that you can: • restart independently • place in different availability zones, racks, whatever • use different hardware if you really wanted to do something weird it makes sense to switch when those become features you need.

In addition, the fan-out part becomes interesting. Any one slow server (e.g. a GC) will cause the entire query to slow down. So, a higher fan-out increases the probability of one server being slow. So, at some point you will want to limit the fan-out and use replica groups instead.