Hello everyone,
What does Pinot store in zookeeper...
# generalp
Pedro Silva
05/11/2021, 3:41 PMHello everyone,
What does Pinot store in zookeeper metadata. I currently have 2GB out of 2.5GB of disk used up (78.5%) in my zookeeper instance. Should this be a cause for concern?
m
Mayank
05/11/2021, 3:42 PM
Cluster state, segment metadata etc. do you have too many segments?
Mayank
05/11/2021, 3:42 PM
Whatever is visible via ZK in the UI is stored inside ZK
p
Pedro Silva
05/11/2021, 3:43 PMWhat is too many segments? I have 1 table only with 3842 segments.
m
Mayank
05/11/2021, 3:44 PM
Yeah 3.8k segments might lead to 2GB, each segment has metadata stored in ZK
Mayank
05/11/2021, 3:45 PM
Too many is relative of course, here I meant enough to justify 2GB disk storage
p
Pedro Silva
05/11/2021, 3:45 PMUnderstood. Can zookeeper information be offloaded to hdfs like the segments?
m
Mayank
05/11/2021, 3:46 PM
Not sure if ZK supports that If it did, then you will incur latency for ZK read/writes