Hey all 👋 We are aiming at having pretty large tables with more than 100TB and we have needed to increase the Zookeeper maxbuffer a couple of times already at a much lower data volume. We have gotten pushback from our ZK infra team about increasing this value since they had some nasty issues with it beforehand. A couple of questions: 1. What is a recommended (often seen) maxbuffer limit? Is there any specific setup of the ZK instance that you have seen before? 2. Is the only way to keep the

IDEALSTATE

and

EXTERNALVIEW

smaller by creating larger segments? It seems like the size of these can also decrease the performance.

1. your zk team is likely right, and it should be low. we’ve set it as high as 20mb and given 27gb of heap to ZK, and it still worked fine. that said, our biggest ZK node at the moment is ~2.7 mb. we also do

-Dzk.serializer.znrecord.auto-compress.enabled=true -Dzk.serializer.znrecord.auto-compress.threshold.bytes=0

to get the node size down 2. having fewer segments by making them larger is the best approach. another common one is to make sure segment names are short (don’t put the full table name if it’s long, don’t put datetimes in the segments name, etc) since for tables with 10k+ segments, the extra bytes add up quickly but if you still hit the limit, there’s no other workaround that i know of. if you don’t increase it, pinot will break

how many segments in that table.

We are currently at 130k

yeah. thats a good thing to do.

Each segment holds around 500k rows currently

500k rows per segments - thats too small. do you have blob's in each row?

We have pretty beefy events unfortunately. I can send you the schema in a PM

Yeah. Will be happy to review and see if there is something we can do in Pinot to optimize