https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
b
Hello Ali, thanks for reaching to the pulsar community. What version of pulsar do you use and what is the output you see when you run the -a -h command..
CC: @Ming
a
I'm testing this on 3.0, however there was a problem on 2.11
while an upgrade was in progress from 2.10, bookies or PVC was deleted
Copy code
$ bin/bookkeeper shell listbookies -a -
.....
2023-07-04T16:44:31,877+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.library.path=/usr/java/packages/lib:/usr/lib64:/lib64:/lib:/usr/lib
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.io.tmpdir=/tmp
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:java.compiler=<NA>
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.name=Linux
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.arch=amd64
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.version=5.4.238-148.347.amzn2.x86_64
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.name=?
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.home=?
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:user.dir=/pulsar
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.free=111MB
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.max=256MB
2023-07-04T16:44:31,878+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Client environment:os.memory.total=130MB
2023-07-04T16:44:31,882+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Initiating client connection, connectString=pulsar-apache-zookeeper:2181 sessionTimeout=30000 watcher=org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase@5be067de
2023-07-04T16:44:31,886+0000 [main] INFO  org.apache.zookeeper.common.X509Util - Setting -D jdk.tls.rejectClientInitiatedRenegotiation=true to disable client-initiated TLS renegotiation
2023-07-04T16:44:31,906+0000 [main] INFO  org.apache.zookeeper.ClientCnxnSocket - jute.maxbuffer value is 1048575 Bytes
2023-07-04T16:44:31,914+0000 [main] INFO  org.apache.zookeeper.ClientCnxn - zookeeper.request.timeout value is 0. feature enabled=false
2023-07-04T16:44:31,924+0000 [main-SendThread(pulsar-apache-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Opening socket connection to server pulsar-apache-zookeeper/10.104.86.163:2181.
2023-07-04T16:44:31,925+0000 [main-SendThread(pulsar-apache-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - SASL config status: Will not attempt to authenticate using SASL (unknown error)
2023-07-04T16:44:31,932+0000 [main-SendThread(pulsar-apache-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Socket connection established, initiating session, client: /10.104.44.32:46836, server: pulsar-apache-zookeeper/10.104.86.163:2181
2023-07-04T16:44:31,940+0000 [main-SendThread(pulsar-apache-zookeeper:2181)] INFO  org.apache.zookeeper.ClientCnxn - Session establishment complete on server pulsar-apache-zookeeper/10.104.86.163:2181, session id = 0x100d33a1a4d0063, negotiated timeout = 30000
2023-07-04T16:44:31,943+0000 [main-EventThread] INFO  org.apache.bookkeeper.zookeeper.ZooKeeperWatcherBase - ZooKeeper client is connected now.
2023-07-04T16:44:32,044+0000 [main-EventThread] INFO  org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-apache-bookie-2.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-apache-bookie-2.pulsar-apache-bookie.pulsar-apache.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-07-04T16:44:32,045+0000 [main-EventThread] INFO  org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-apache-bookie-0.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-apache-bookie-0.pulsar-apache-bookie.pulsar-apache.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-07-04T16:44:32,045+0000 [main-EventThread] INFO  org.apache.bookkeeper.discover.ZKRegistrationClient - Update BookieInfoCache (writable bookie) pulsar-apache-bookie-3.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181 -> BookieServiceInfo{properties={}, endpoints=[EndpointInfo{id=httpserver, port=8000, host=0.0.0.0, protocol=http, auth=[], extensions=[]}, EndpointInfo{id=bookie, port=3181, host=pulsar-apache-bookie-3.pulsar-apache-bookie.pulsar-apache.svc.cluster.local, protocol=bookie-rpc, auth=[], extensions=[]}]}
2023-07-04T16:44:32,055+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - All Bookies :
2023-07-04T16:44:32,062+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-apache-bookie-2.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181, IP:10.104.44.32, Port:3181, Hostname:pulsar-apache-bookie-2.pulsar-apache-bookie.pulsar-apache.svc.cluster.local
2023-07-04T16:44:32,062+0000 [main] INFO  org.apache.bookkeeper.client.DefaultBookieAddressResolver - Cannot resolve pulsar-apache-bookie-1.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181, bookie is unknown org.apache.bookkeeper.client.BKException$BKBookieHandleNotAvailableException: Bookie handle is not available
2023-07-04T16:44:32,063+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-apache-bookie-1.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181, IP:UNKNOWN, Port:0, Hostname:UNKNOWN
2023-07-04T16:44:32,064+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-apache-bookie-0.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181, IP:10.104.95.113, Port:3181, Hostname:pulsar-apache-bookie-0.pulsar-apache-bookie.pulsar-apache.svc.cluster.local
2023-07-04T16:44:32,065+0000 [main] INFO  org.apache.bookkeeper.tools.cli.commands.bookies.ListBookiesCommand - BookieID:pulsar-apache-bookie-3.pulsar-apache-bookie.pulsar-apache.svc.cluster.local:3181, IP:10.104.81.102, Port:3181, Hostname:pulsar-apache-bookie-3.pulsar-apache-bookie.pulsar-apache.svc.cluster.local
2023-07-04T16:44:32,170+0000 [main] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x100d33a1a4d0063 closed
2023-07-04T16:44:32,170+0000 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x100d33a1a4d0063
[1.409s][info ][gc,heap,exit] Heap
[1.409s][info ][gc,heap,exit]  garbage-first heap   total 133120K, used 30410K [0x00000000f0000000, 0x0000000100000000)
[1.409s][info ][gc,heap,exit]   region size 1024K, 22 young (22528K), 8 survivors (8192K)
[1.409s][info ][gc,heap,exit]  Metaspace       used 17854K, committed 18112K, reserved 1064960K
[1.409s][info ][gc,heap,exit]   class space    used 2066K, committed 2176K, reserved 1048576K
m
If you have auto recovery pod set up, it should be able to rebuild ledgers from the new empty pvc
I realize you said the PVC is deleted. Is the PV deleted? If you have the PV’s persistentVolumeReclaimPolicy set as 
Retain
or it’s not deleted, you can bind the PVC to the retained PV. The pv is the volumename in the pvc manifest file. There is no need to all the above steps.
Open in Slack
PreviousNext
Powered by