Capture3.PNG
# troubleshootings
Sadim Nadeem
08/30/2021, 8:57 AMCapture3.PNG
Sadim Nadeem
08/30/2021, 9:43 AM@Xiang Fu
Sadim Nadeem
08/30/2021, 10:24 AMwhat is this sessionId does not match about cc: @Mayank @Xiang Fu @Kishore G @Jackie @Subbu Subramaniam
x
Xiang Fu
08/30/2021, 6:05 PM
it comes from the process restart
Xiang Fu
08/30/2021, 6:07 PM
the previous session is still in zookeeper, which will be cleaned after zookeeper session timeout
s
Sadim Nadeem
08/30/2021, 6:52 PMhow to resolve this issue ..
j
Jackie
08/30/2021, 6:56 PMThis is normal during broker restart/reconnect, and you may ignore them
s
Sadim Nadeem
08/30/2021, 6:58 PMi think restarting broker alone could have resolve the broker crashloopbackoff issue .. the mistake we were making was restarting everything including controller , server .. I guess we got the trick now ..
Sadim Nadeem
08/30/2021, 6:59 PMor the other solution would be increasing the health check timeout .. not sure how to increase it
Sadim Nadeem
08/30/2021, 7:00 PMwe faced the same issue around a month ago .. I think same issue faced again 1 month later today but forgot the resolution .. now I can recollect
x
Xiang Fu
08/30/2021, 8:25 PM
I think the issue is still why broker will fail, if you can provide the logs of failed broker, it can help us find the issue
s
Sadim Nadeem
08/31/2021, 7:52 AMplease check if it help[s.. its a partial log .. will check if can get the whole log .. most probably its not present
x
Xiang Fu
08/31/2021, 8:17 AM
From the log, seems the issue is on zookeeper
Xiang Fu
08/31/2021, 8:17 AM
can you check the zookeeper logs for the same time period?
s
Sadim Nadeem
08/31/2021, 8:18 AMok let me check
x
Xiang Fu
08/31/2021, 8:18 AM
also, is there any zookeeper restart or downtime during that time?
s
Sadim Nadeem
08/31/2021, 8:19 AMno .. zookeeper never restarted
Sadim Nadeem
08/31/2021, 8:19 AMyou can check here .. most of the time zk has 0 restarts
x
Xiang Fu
08/31/2021, 9:23 AM
there is one restart:
s
Sadim Nadeem
08/31/2021, 9:23 AMyes
Sadim Nadeem
08/31/2021, 9:24 AMbut we have 3 zk nodes for high availability
Sadim Nadeem
08/31/2021, 9:24 AMso restart of one pod shouldnt cause much trouble
x
Xiang Fu
08/31/2021, 9:51 AM
true, please try to correlate this issue with a zookeeper log if you found it next time
Xiang Fu
08/31/2021, 9:51 AM
also you can use kubectl cmd to delete broker pod to let it restart
s
Sadim Nadeem
08/31/2021, 9:53 AMsure @Xiang Fu
Sadim Nadeem
09/01/2021, 6:11 PMwhat are the things that consume zookeeper disk space .. I can see disk space utilization increasing by 1`-2% every day .. in next 1 month it will get full .. so was curious to know what are the things that consumes zk space in pinot @Xiang Fu @Jackie @Mayank
m
Mayank
09/01/2021, 6:31 PM
I am guessing it is the ZK snapshots (or logs)?
s
Sadim Nadeem
09/01/2021, 9:10 PMis there some way to get rid of old logs automatically from disk so that zk dont crashes in future bcz of disk space issue
m
Mayank
09/01/2021, 10:20 PM
I am guessing there should be. You want to check ZK docs?
s
Sadim Nadeem
09/02/2021, 10:09 AMdidnt got much help https://docs.pinot.apache.org/basics/architecture#apache-helix-and-zookeeper