Hi, we are running pulsar 3.0.12 in production, k8s. (zookeeper 3.9.2) We noticed that today zookeeper logs suddenly started showing WARN logs like: 112535.054 [CommitProcessor:2] WARN org.apache.zookeeper.server.DataTree - Message:Digests are not matching. Value is Zxid. Value:240759533065 112535.090 [CommitProcessor:2] WARN org.apache.zookeeper.server.DataTree - Message:Digests are not matching. Value is Zxid. Value:240759533068 112535.112 [CommitProcessor:2] WARN org.apache.zookeeper.server.DataTree - Message:Digests are not matching. Value is Zxid. Value:240759533073 It happened to coincide with bookie rollout we did today. Not sure how that is related at all. Has anyone seen this error before? Any action that I should be taking? Search results have not been helpful for this error unfortunately. Thanks in advance.

I just searched the Jira issues for "Digests are not matching": https://issues.apache.org/jira/browse/ZOOKEEPER-4357?jql=project%20%3D%20ZOOKEEPER%20[…]0text%20~%20%22%5C%22Digests%20are%20not%20matching%5C%22%22 . No clear answer there. Just curious: which deployment solution do you use for Zookeeper? Apache Pulsar Helm chart? One detail to check is to ensure that Zookeeper service has

publishNotReadyAddresses: true

. In old helm chart versions it used

<http://service.alpha.kubernetes.io/tolerate-unready-endpoints|service.alpha.kubernetes.io/tolerate-unready-endpoints>

annotation, which was removed completely in k8s 1.24. It might not be related to this issue at all, but just came into mind.

Thanks for the response @Lari Hotari. So, the warning persisted for 4 hours and went away on its own. Weird 😅. I will look into this more, and will update if I get any further here. At this point seems like some bug around how digest is being calculated by leader zk.