Is there any issue with having a large number of tables on a single cluster in terms of zk latency, metadata size, etc.? We currently have ~75 tables.

we have 4000+ tables.

Thanks:) Could it be constant uploading of offline segments that increases zk latency - we've only seen this issue in the past month.

You can choose to split the pinot and helix controllers. It has helped us. You should increase the network speed between zk and controllers as well.

you mean use split commit? or is this something else?

@Jack do we have a doc on how to separate the pinot and helix controller hardware ? We do have the steps in the design doc, but much of that is verified (when we did it, we had to make sure that we can also come back out of the split mode. You don't need to worry about it, since we have tested it for several years now).

thanks so much @Subbu Subramaniam!

Right, that’s the correct doc. 👆 Feel free to try it out! :)

ok:) Thanks @Jack!

I dont think splitting is needed at 75 tables

thanks

Elon, are you using metadata uri push or data push?

Still using data push and for most tables we're still not using peer download (just started to roll it out)

do you really know its ZK network? do you have graphs?

Yes - for the past 2 months the zk network io has been steadily increasing

Then you might be having too many small segments issue

Nice, that sounds like it's the issue! We have many realtime tables that are very small segments - we use daily push frequency, should we also have max time set to 24 hours?

how many segments is too many?

and is it the amount of segments or the size?

I was thinking overnight and arrived at the same conclusion that you shoul dnot need to split at 75. We split when were at 2000 or so. Sounds like you have identified the problem to be that of small segments. Check why you have such small segments. Is it tht your inbound stream is too slow? Can you reduce the number of partitions? Or, can you change the time to 48 hours so that larger segmengs can be made? I hope you are using the segment size option in the realtime config

Thanks!

oh, you are using merge/rollup to combine segments? We use it in our 4000-table cluster, and have not seen any problems so far. @Seunghyun any comments?

nice:) We're increasing zk session timeouts to see if that helps

e did have problems with timeouts earlier, and we set it to a larger value. I will let @Seunghyun help u here

thanks @Subbu Subramaniam!

@Elon If your ZK is too slow and make it work the

endReplaceSegments

, there’s a config to set the timeout for

endReplacement

on the minion side

nice! I can take a look but would you know off the top of your head which code I can look at?

Please refer

MinionConf

. There’s the config called

pinot.minion.endReplaceSegments.timeoutMs

Thanks! Really appreciate the help, this is the best community:) I promise to make some contributions back:)

@Elon which issue can i assign to you ? 🙂

any that you want! lmk I would be super happy to help in any way I can:)

By curiosity, are you trying out merge/rollup?

we are but still using pinot-0.8.0

FYI, we don’t support realtime tables yet

I thought offline

it will work with offline.

ah,you mean it converts to offline, got it

it should be a different discussion, but I think Pinot should fail before is silently get's into inconsistent state

inconsistent meaning: segments in gcs not downloaded to servers, or replicas have different versions of a segment (different crc), or replication factor < desired for a certain amount of time.