I can see the pinot-zookeeper disk usage high .. w...
# troubleshootings
Sadim Nadeem
09/07/2021, 8:59 AMI can see the pinot-zookeeper disk usage high .. what could be the root cause since metadata cant be 65 gb out of 95 gb given with few million records in table .. is it database indexing causing some external views to be stored on zookeeper disk? @Mayank @Xiang Fu @Jackie @Subbu Subramaniam @Kishore G
p
Peter Pringle
09/07/2021, 12:28 PMYou need to setup a zookeeper clean up job. To remove old snapshots.
Peter Pringle
09/07/2021, 12:30 PMm
Mayank
09/07/2021, 1:05 PM
Thanks @Peter Pringle
s
Sadim Nadeem
09/10/2021, 6:46 PM@Xiang Fu pinot brokers again went to crashloopback status once zookeeper disk storage got full.. Do you have any input what is causing zookeeper disk space to be exhausted and how to clean it and if zookeeper is full.. Why pinot brokers crash and how to bring pinot back to running stage with all tables and data intact if such a situation happens after cleaning zookeeper Or restoring it from attached pvc on google kubernetes engine
Sadim Nadeem
09/10/2021, 6:47 PMWe end up losing all the tables and data when this problem happens
Sadim Nadeem
09/10/2021, 6:47 PMAlready there is a thread with description and log of the same issue which replicated again and we are getting regression of this issue
Sadim Nadeem
09/10/2021, 6:48 PMSadim Nadeem
09/10/2021, 6:48 PMhttps://apache-pinot.slack.com/files/U0224CF6LMT/F02CK3E6EKG/capture3.png▾
x
Xiang Fu
09/10/2021, 7:15 PM
you can config environment variables :
ZK_SNAP_RETAIN_COUNT=3ZK_PURGE_INTERVAL=1s
Sadim Nadeem
09/11/2021, 11:53 AMOk will try that and update if issue regression happenS further
m
Mayank
09/15/2021, 7:11 PM
@beerus ^^
s
Sadim Nadeem
09/27/2021, 7:53 AM@Xiang Fu which file do we need to config these variables in helm config :
ZK_SNAP_RETAIN_COUNT=3ZK_PURGE_INTERVAL=1Sadim Nadeem
09/27/2021, 7:55 AMwe are not aware of this cc: @Mohamed Kashifuddin @Shailesh Jha @Mohamed Hussain
x
Xiang Fu
09/27/2021, 8:03 AM
It’s environment variables, so check values.yaml
s
Sadim Nadeem
09/27/2021, 8:19 AMdo we need to add it here @Xiang Fu in pinot/values.yaml?
Sadim Nadeem
09/27/2021, 8:58 AM@Xiang Fu I am worried if setting zk snap retention period in ZK might have some consequence. AS it store Metadata. about all the tables and data.
will there be some consequences?(means attaching the pvc back when restarting pinot pods after helm upgrade .. will tables data remain intact?)
Because If We set retention as 3 day and metadata cleaned. What about older meta data / zk znodes?? can you enlighten a bit
Sadim Nadeem
09/27/2021, 10:05 AMor need to add here ?
x
Xiang Fu
09/27/2021, 3:44 PM
Yes
s
Sadim Nadeem
09/27/2021, 4:14 PMWhat about the consequences for adding these configs https://apache-pinot.slack.com/archives/C011C9JHN7R/p1632733114427600?thread_ts=1631005184.110500&cid=C011C9JHN7R
x
Xiang Fu
09/27/2021, 5:08 PM
this retention is for zk snapshots, your data are still full data.
🍷 1
Xiang Fu
09/27/2021, 5:09 PM
s
Sadim Nadeem
09/27/2021, 6:07 PMThanks @Xiang Fu
Sadim Nadeem
10/10/2021, 7:14 PMHi @Xiang Fu..even after adding these environment variables and updating helm ..the disk usage keeps on increasing .. do i also need to do
kubectl delete po dev-pinot-zookeeper-0 dev-pinot-zookeeper-1 dev-pinot-zookeeper-2 -n dev-pinot in order to get the environment variable changes to actually work for setting up cron job to clean up old snapshots and only helm upgrade is not enough ?
x
Xiang Fu
10/11/2021, 3:57 AM
then it means you are not setting it correctly, please read the script to start zookeeper and figure out the environment variable name
s
Sadim Nadeem
10/11/2021, 4:43 AMok i will go through http://zookeeper.apache.org/doc/r3.1.2/zookeeperAdmin.html#sc_maintenance
x
Xiang Fu
10/11/2021, 4:52 AM
also enter into the k8s pod to check the start script to see how those configs are set
s
Sadim Nadeem
10/11/2021, 4:52 AMsure @Xiang Fu .. will check and update
l
Luis Fernandez
04/18/2022, 7:55 PMhey Sadim did you figure how to fix this? We are seeing kinda the same issue with our zookeeper images going up in size
s
Shailesh Jha
07/06/2022, 4:20 PMEnabling autopurge in zookeeper config should resolve this issue @Luis Fernandez
👍 2