https://pulsar.apache.org/ logo
Join Slack
Powered by
Hi team: I DO REALLY NEED THE HELP RIGHT NOW. One ...
# general
b
Hi team: I DO REALLY NEED THE HELP RIGHT NOW. One of my bookie node in the production cluster are stuck in the CrashLoopBackOff status; after checking the logs, I find out that it is because the node cannot successfully access the lock from the metadata server (which we use Oxia in our cluster). I have tried to restart the Oxia cluster, since I saw some error messages saying that the oxia cannot append the WAL to the stream (I were facing the similar situation last month, and the restarting Oxia trick did work for me). Unfortunately, the bookie node still cannot get it's own lock after restarting the Oxia. I REALLY NEED SOMEONE THAT CAN HELP ME OUT WITH THE ISSUE 😢
one of the error log I found is
Copy code
{
  "textPayload": "java.util.concurrent.ExecutionException: org.apache.pulsar.metadata.api.MetadataStoreException$LockBusyException: Resource at /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar-bilanx.svc.cluster.local:3181 is already locked\n\tat java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.CompletableFuture.get(Unknown Source) ~[?:?]\n\tat org.apache.pulsar.metadata.bookkeeper.PulsarRegistrationManager.registerBookie(PulsarRegistrationManager.java:159) ~[org.apache.pulsar-pulsar-metadata-4.0.5.jar:4.0.5]\n\tat org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:330) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]\n\tat org.apache.bookkeeper.bookie.BookieStateManager.doRegisterBookie(BookieStateManager.java:318) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]\n\tat org.apache.bookkeeper.bookie.BookieStateManager$3.call(BookieStateManager.java:281) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]\n\tat org.apache.bookkeeper.bookie.BookieStateManager$3.call(BookieStateManager.java:276) ~[org.apache.bookkeeper-bookkeeper-server-4.17.1.jar:4.17.1]\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) ~[?:?]\n\tat java.base/java.lang.Thread.run(Unknown Source) [?:?]\nCaused by: org.apache.pulsar.metadata.api.MetadataStoreException$LockBusyException: Resource at /ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar-bilanx.svc.cluster.local:3181 is already locked\n\tat org.apache.pulsar.metadata.coordination.impl.ResourceLockImpl.lambda$revalidate$19(ResourceLockImpl.java:328) ~[org.apache.pulsar-pulsar-metadata-4.0.5.jar:4.0.5]\n\tat java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source) ~[?:?]\n\tat java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source) ~[?:?]\n\tat io.streamnative.oxia.client.batch.Operation$ReadOperation$GetOperation.complete(Operation.java:77) ~[io.streamnative.oxia-oxia-client-0.5.0.jar:?]\n\tat io.streamnative.oxia.client.batch.ReadBatch.onNext(ReadBatch.java:73) ~[io.streamnative.oxia-oxia-client-0.5.0.jar:?]\n\tat io.streamnative.oxia.client.batch.ReadBatch.onNext(ReadBatch.java:29) ~[io.streamnative.oxia-oxia-client-0.5.0.jar:?]\n\tat io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:478) ~[io.grpc-grpc-stub-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInternal(ClientCallImpl.java:660) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:647) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.SerializeReentrantCallsDirectExecutor.execute(SerializeReentrantCallsDirectExecutor.java:49) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.messagesAvailable(ClientCallImpl.java:675) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.ForwardingClientStreamListener.messagesAvailable(ForwardingClientStreamListener.java:39) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.AbstractStream$TransportState.messagesAvailable(AbstractStream.java:183) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.MessageDeframer.processBody(MessageDeframer.java:413) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.MessageDeframer.deliver(MessageDeframer.java:276) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.MessageDeframer.deframe(MessageDeframer.java:178) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.AbstractStream$TransportState.deframe(AbstractStream.java:211) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.AbstractClientStream$TransportState.inboundDataReceived(AbstractClientStream.java:360) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.internal.Http2ClientStreamTransportState.transportDataReceived(Http2ClientStreamTransportState.java:148) ~[io.grpc-grpc-core-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.grpc.netty.NettyClientStream$TransportState.transportDataReceived(NettyClientStream.java:341) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.onDataRead(NettyClientHandler.java:394) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler.access$1100(NettyClientHandler.java:93) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.grpc.netty.NettyClientHandler$FrameListener.onDataRead(NettyClientHandler.java:923) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder$FrameReadListener.onDataRead(DefaultHttp2ConnectionDecoder.java:307) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger$1.onDataRead(Http2InboundFrameLogger.java:48) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readDataFrame(DefaultHttp2FrameReader.java:415) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.processPayloadState(DefaultHttp2FrameReader.java:250) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2FrameReader.readFrame(DefaultHttp2FrameReader.java:159) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2InboundFrameLogger.readFrame(Http2InboundFrameLogger.java:41) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.DefaultHttp2ConnectionDecoder.decodeFrame(DefaultHttp2ConnectionDecoder.java:173) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler$FrameDecoder.decode(Http2ConnectionHandler.java:393) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.http2.Http2ConnectionHandler.decode(Http2ConnectionHandler.java:453) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\tat io.grpc.netty.shaded.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.grpc-grpc-netty-shaded-1.56.1.jar:1.56.1]\n\t... 1 more",
  "insertId": "xqh65pgzckolj4wg",
  "resource": {
    "type": "k8s_container",
    "labels": {
      "location": "asia-east1",
      "pod_name": "pulsar-bookie-0",
      "namespace_name": "pulsar-bilanx",
      "container_name": "pulsar-bookie",
      "project_id": "dcard-production",
      "cluster_name": "event-1"
    }
  },
  "timestamp": "2025-10-17T02:53:55.643835022Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/release": "pulsar",
    "k8s-pod/apps_kubernetes_io/pod-index": "0",
    "<http://logging.gke.io/top_level_controller_name|logging.gke.io/top_level_controller_name>": "pulsar-bookie",
    "<http://compute.googleapis.com/resource_name|compute.googleapis.com/resource_name>": "gke-event-1-pulsar-1-b7130055-2kjr",
    "k8s-pod/app": "pulsar",
    "k8s-pod/statefulset_kubernetes_io/pod-name": "pulsar-bookie-0",
    "k8s-pod/cluster": "pulsar-bilanx",
    "k8s-pod/component": "bookie",
    "k8s-pod/kosko_dev/component": "pulsar-bilanx",
    "<http://logging.gke.io/top_level_controller_type|logging.gke.io/top_level_controller_type>": "StatefulSet",
    "k8s-pod/controller-revision-hash": "pulsar-bookie-57d6dffdfc"
  },
  "logName": "projects/dcard-production/logs/stdout",
  "receiveTimestamp": "2025-10-17T02:53:57.164293353Z"
}
m
Hi @benjamin99
what is your oxia server version?
You could delete the key 
/ledgers/available/pulsar-bookie-0.pulsar-bookie.pulsar-bilanx.svc.cluster.local:3181
on the oxia side and then restart bookie as a workaround.
the issue should be fixed in this PR. https://github.com/oxia-db/oxia/pull/742
created a channel for oxia. #C09LL9Y8JCX
b
Yes. I just did that by using the client command in one of the oxia pod in the cluster, and it seems to work since the bookie can successfully init now.
👍 1
m
What is the Oxia version you are using?
b
the image is: 
streamnative/oxia:0.12.0
so I believe it would be 0.12.0
the is should be setup by the helm chart with version: 4.1.0
m
You should go for 
oxia-db/oxia:0.14.4
, which includes the fix. 🙂
b
sounds good. but is it ok for me to run the in place update from v0.12.0 to v0.14.4?
m
Yes, that should be compatible. 🙂
but for pulsar-side
you have to upgrade the oxia-client-java to >= 0.6.2
b
so which means I would have to upgrade all pulsar components to a specific version?
m
Yep, you can go for the latest pulsar version from v4.0 🙂
Yes
b
Good. I will try to update both pulsar and the oxia latter for resolving the locking issue of the bookies.
👍 1
l
FYI, Apache Pulsar Helm chart release 4.3.0 bumped Oxia version to 0.14.4.
2 Views
Open in Slack
PreviousNext
Powered by