Hi All, I am facing one issue in pulsar 4.0. One of my Zookeeper pods getting terminated after 4 or 5 days and not recovering . Reason for not recovering is k8s node zookeeper is running on has reached max mount point i.e 16384. My k8s node is also storage node(Oracle Block Volume). We have not faced such issue while we were on pulsar 3.1.1.

It would be better to ask this question at https://github.com/apache/pulsar/discussions/new?category=q-a and link the question here. It would be useful to cover answers to these questions when you ask the question • What way do you deploy to Kubernetes? ◦ If you are using Apache Pulsar Helm chart, what is the Helm chart version? • What is the Apache Pulsar release version? ◦ For example 4.0.0 and 4.0.1 contain an issue that is fixed since 4.0.2 with https://github.com/apache/pulsar/pull/23667 . Please ensure that your are using Pulsar 4.0.8 . • Why does the Zookeeper pod get terminated? ◦ Is it due to memory (OOM Killed etc, OutOfMemoryError)? • Have you already increased the defaults? Key is

zookeeper.configData.PULSAR_MEM

in the values.yaml file. The default

-Xms64m -Xmx128m

is very low. You could bump it to

-Xms256m -Xmx512m

• One of the significant differences made in the Pulsar Helm chart defaults is the switch from G1GC to ZGC. It's possible that ZGC uses slightly more RAM than G1GC. This change is included since Helm chart version 4.3.0 (commit: https://github.com/apache/pulsar-helm-chart/commit/6f1585e65e67c8890b7973e8fd574fd1b990101f).

Thanks a lot for your reply. I already removed timeout command from readiness probe and liveness probe last week(Wednesday). But today again zookeeper pods got restarted and did not recover. My zookeeper is running with 7Gb of Xms and Xmx and memory usage is almost 4.5 GB. My pulsar version is 4.0.0 and still using G1GC. below is the configuration used in configMap data: dataDir: /pulsar/data/zookeeper PULSAR_PREFIX_serverCnxnFactory: org.apache.zookeeper.server.NIOServerCnxnFactory serverCnxnFactory: org.apache.zookeeper.server.NIOServerCnxnFactory PULSAR_GC: | -XX:+UseG1GC -XX:MaxGCPauseMillis=25 -Dcom.sun.management.jmxremote -Djute.maxbuffer=10485760 -XX:+ParallelRefProcEnabled -XX:+UnlockExperimentalVMOptions -XX:+DoEscapeAnalysis -XX:+DisableExplicitGC -XX:+ExitOnOutOfMemoryError -XX:+PerfDisableSharedMem PULSAR_MEM: | -Xms7168m -Xmx7168m ------

Why are you running Pulsar 4.0.0 ? There are a lot (hundreds) of bug fixes since it was released. I'd recommend upgrading to Pulsar 4.0.8 images.

Reason so far we found is timeout command. It was creating 65535 child process. and container was terminating. We removed timeout command but still it got restarted today. This the o/p of describe command

Name:                      zookeeper-1
Namespace:                 integration
Priority:                  0
Service Account:           zookeeper
Node:                      172.19.6.139/172.19.6.139
Start Time:                Thu, 12 Feb 2026 04:01:37 +0530
Labels:                    app=pulsar
                           cluster=integration-pulsar
                           component=zookeeper
                           controller-revision-hash=zookeeper-75b4cc68f8
                           harness.io/release-name=release-2b5e4a
                           statefulset.kubernetes.io/pod-name=zookeeper-1
Annotations:               prometheus.io/port: 8000
                           prometheus.io/scrape: false
Status:                    Terminating (lasts 93m)
Termination Grace Period:  30s
IP:                        10.103.16.253
IPs:
  IP:           10.103.16.253
Controlled By:  StatefulSet/zookeeper
Containers:
  pulsar-zookeeper:
    Container ID:  <cri-o://c5e706134f3fb9d4f049f1e3b3c46ce05c7b63bf26472f56be090c4af1708ce>b
    Image:         ghcr.io/xcorp/pulsar-all:4.0.0
    Image ID:      3ed45720a8bbffc60d15310cbe8d482c695cd68af8f2bd9e145c1939fd2f0030
    Ports:         8000/TCP, 2181/TCP, 2888/TCP, 3888/TCP
    Host Ports:    0/TCP, 0/TCP, 0/TCP, 0/TCP
    Command:
      sh
      -c
    Args:
      bin/apply-config-from-env.py conf/zookeeper.conf;
      bin/generate-zookeeper-config.sh conf/zookeeper.conf; OPTS="${OPTS} -Dlog4j2.formatMsgNoLookups=true" exec bin/pulsar zookeeper;

    State:          Terminated
      Reason:       Error
      Exit Code:    143
      Started:      Thu, 12 Feb 2026 04:01:45 +0530
      Finished:     Mon, 16 Feb 2026 17:09:42 +0530
    Ready:          False
    Restart Count:  0
    Requests:
      cpu:      4
      memory:   7Gi
    Liveness:   exec [sh -c { echo ruok; sleep 1; } | nc 127.0.0.1 2181 | grep imok] delay=25s timeout=30s period=30s #success=1 #failure=10
    Readiness:  exec [sh -c { echo ruok ; sleep 1; } | nc 127.0.0.1 2181 | grep imok] delay=25s timeout=30s period=30s #success=1 #failure=10
    Environment Variables from:
      zookeeper  ConfigMap  Optional: false
    Environment:
      ZOOKEEPER_SERVERS:          zookeeper-0,zookeeper-1,zookeeper-2
      EXTERNAL_PROVIDED_SERVERS:  false
    Mounts:
      /pulsar/data from data-vol (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-xf4sb (ro)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  data-vol:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  data-vol-zookeeper-1
    ReadOnly:   false
  kube-api-access-xf4sb:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   Burstable
Node-Selectors:              <none>
Tolerations:                 node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                             node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:                      <none>

This mount issue was there when we were using timeout command

If that was the case, it was already fixed in Pulsar 4.0.2 docker images with https://github.com/apache/pulsar/pull/23667

We have not set limit for memory. K8s node is having 256 Gi of RAM. We generally dont set limit for conatainer. For pulsar 3.1.1, we have given 4 GB and zookeeper was working without any issue for atleast 2 years. After pulsar 4.0.0 upgrade , we saw memory usage increasing to 5.5 GB few times so we increased to 7Gb. Also , No. Of topics / producer/ consumer has increased now by 5 times. We will upgrade to 4.0.8 and let you know. Thanks again for your timely response.

We have not set limit for memory. K8s node is having 256 Gi of RAM.

The problem I'm pointing to is that when the request is lower than the actual consumption, it could cause issues at the node level when the node's RAM is fully occupied. A JVM will use more memory than the max heap size. If you set the max heap size to 7GB, you should set requests to about 10Gi to ensure that the pod doesn't get scheduled on a node that is full.

ohh

You could use the Zookeeper Grafana dashboard to look into more specific metrics.