Could events like deleting a large Pinot table (with ~10k segments) generate a lot of messages to servers and results in a lot of nodes in Helix/Zk?

yes

If some servers are down, will the message nodes be deleted?

it will be deleted once the nodes come back up

some of our nodes can not get back up because

WARN org.apache.zookeeper.ClientCnxn - Session 0x3678639af2315ed for server zknode/10.80.14.25:2181, unexpected error, closing socket connection and attempting reconnect java.io.IOException: Packet len6418980 is out of range!

at org.apache.zookeeper.ClientCnxnSocket.readLength(ClientCnxnSocket.java:113) at org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:79) at org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:366) at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1144)

Could there be too many messages for a server?

whats the zookeeper max buffer?

now or was it like that before

increased it to 10MB

on the pinot node side or zookeeper server

pinot side

client buffer

I think this is server side config

hmm. but the exception is from

ClientCnxnSocket.readLength

whats the setting on server

zookeeper server

i vaguely remember you needed this on the zookeeper server as well

this is the zk server setting

ok, that seems to be fine

ok

need a quick review on this

I had left a question on the PR (I am very likely missing something obvious)

what was the questio

Neha just replied

ok

anything else needed

No, just e2e test as others also mentioned.