This message was deleted pulsar #general

Join Slack

This message was deleted.

# general

Slackbot

07/06/2023, 10:03 AM

This message was deleted.

Asaf Mesika

07/06/2023, 10:10 AM

@Hang Chen is direct memory king recently 🙂

🙂 1

Hang Chen

07/06/2023, 10:24 AM

It looks like this PR fix it https://github.com/apache/bookkeeper/pull/3746

Hang Chen

07/06/2023, 10:27 AM

Could you use BookKeeper 4.14.7 to test whether this issue still exists? BookKeeper 4.14.6 and 4.14.7 fixed a lot of memory leak issues https://github.com/apache/bookkeeper/releases/tag/release-4.14.7 https://github.com/apache/bookkeeper/releases/tag/release-4.14.6

Debashish Mallick

07/06/2023, 11:05 AM

@Asaf Mesika @Hang Chen Thanks a lot for giving this info . I will check on this and get back.

Lari Hotari

07/06/2023, 11:09 AM

@Debashish Mallick Is there a specific reason to remain on Pulsar 2.8.2 ? We have fixed a lot of severe security issues and memory leaks after that. Also in Pulsar. Security advisories: https://pulsar.apache.org/security/

Debashish Mallick

07/06/2023, 11:42 AM

@Lari Hotari Thanks for your response . Earlier our product were using 2.8.2 pulsar , Few month back we moved to pulsar 2.10.3 which has bookie 4.14.5. May be we have to cherry pick pulsar PR's for both 2.10.3 and 2.8.2. Could I request you to suggest which PR is required to cherry pick for pulsar.

Lari Hotari

07/06/2023, 12:43 PM

Could I request you to suggest which PR is required to cherry pick for pulsar.

Just curious, why do you use this type of maintenance process? There are a lot of bugs that have been fixed and if you run on an old version of Pulsar, the probability that you hit an already fixed issue is high. Without understanding the context, I'd recommend keeping your Pulsar version as close as possible to the maintained versions. It is true that there is risk for regressions when upgrading, but if you would be doing that early, the bugs would come to the surface and get fixed in the opensource project earlier.

Debashish Mallick

07/07/2023, 9:58 AM

@Hang Chen We upgraded bookie version 4.14.3(pulsar 2.8.2) to 4.14.7 (pulsar 2.10.3) and used used pulsar 2.10.4 which has 4.14.7 bookie , looks like same issue still persist 😞 .Basically I am upgrading from pulsar 2.8.2 to 2.10.4 pulsar with existing journal and ledgers. Is it a preventive solution or we can recover from DirectMemory error?

Copy code

2023-07-07T09:51:06,998 [main] INFO  org.apache.bookkeeper.bookie.Bookie - Replaying journal 1644749554714 from position 708935680
2023-07-07T09:51:07,002 [main] INFO  org.apache.bookkeeper.bookie.JournalChannel - Opening journal data/bookkeeper/journal/current/17ef2b6841a.txn
2023-07-07T09:51:09,162 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Failed to start Component: bookie-server
java.lang.OutOfMemoryError: Cannot reserve 1601533033 bytes of direct buffer memory (allocated: 554639373, limit: 1073741824)
        at java.nio.Bits.reserveMemory(Bits.java:178) ~[?:?]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[?:?]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[?:?]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:243) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:293) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:273) ~[?:?]
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:232) ~[?:?]
        at org.apache.bookkeeper.bookie.JournalChannel.read(JournalChannel.java:261) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Journal.fullRead(Journal.java:1199) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:838) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:997) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:962) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1017) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[com.google.guava-guava-31.0.1-jre.jar:?]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.Main.doMain(Main.java:234) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.Main.main(Main.java:208) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
2023-07-07T09:51:09,173 [main] ERROR org.apache.bookkeeper.common.component.AbstractLifecycleComponent - Calling uncaughtExceptionHandler
2023-07-07T09:51:09,173 [main] ERROR org.apache.bookkeeper.common.component.ComponentStarter - Triggered exceptionHandler of Component: bookie-server because of Exception in Thread: Thread[main,5,main]
java.lang.OutOfMemoryError: Cannot reserve 1601533033 bytes of direct buffer memory (allocated: 554639373, limit: 1073741824)
        at java.nio.Bits.reserveMemory(Bits.java:178) ~[?:?]
        at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:121) ~[?:?]
        at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:332) ~[?:?]
        at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:243) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:293) ~[?:?]
        at sun.nio.ch.IOUtil.read(IOUtil.java:273) ~[?:?]
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:232) ~[?:?]
        at org.apache.bookkeeper.bookie.JournalChannel.read(JournalChannel.java:261) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Journal.fullRead(Journal.java:1199) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Journal.scanJournal(Journal.java:838) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.replay(Bookie.java:997) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.readJournal(Bookie.java:962) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.bookie.Bookie.start(Bookie.java:1017) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.proto.BookieServer.start(BookieServer.java:156) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.service.BookieService.doStart(BookieService.java:68) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:83) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422) ~[com.google.guava-guava-31.0.1-jre.jar:?]
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85) ~[org.apache.bookkeeper-bookkeeper-common-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.Main.doMain(Main.java:234) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
        at org.apache.bookkeeper.server.Main.main(Main.java:208) ~[org.apache.bookkeeper-bookkeeper-server-4.14.7.jar:4.14.7]
2023-07-07T09:51:09,179 [component-shutdown-thread] INFO  org.apache.bookkeeper.common.component.ComponentStarter - Closing component bookie-server in shutdown hook.
2023-07-07T09:51:09,222 [vert.x-eventloop-thread-0] INFO  org.apache.bookkeeper.http.vertx.VertxHttpServer - Starting Vertx HTTP server on port 8000
2023-07-07T09:51:09,231 [component-shutdown-thread] INFO  org.apache.zookeeper.ClientCnxnSocketNetty - channel is told closing
2023-07-07T09:51:09,237 [epollEventLoopGroup-17-1] INFO  org.apache.zookeeper.ClientCnxnSocketNetty - channel is disconnected: [id: 0x331b88c4, L:/172.16.2.41:45130 ! R:itomdipulsar-zookeeper/172.16.1.21:2281]
2023-07-07T09:51:09,242 [component-shutdown-thread] INFO  org.apache.zookeeper.ZooKeeper - Session: 0x3009a1951ba0031 closed
2023-07-07T09:51:09,242 [epollEventLoopGroup-17-1] INFO  org.apache.zookeeper.ClientCnxnSocketNetty - channel is told closing
2023-07-07T09:51:09,244 [main-EventThread] INFO  org.apache.zookeeper.ClientCnxn - EventThread shut down for session: 0x3009a1951ba0031
2023-07-07T09:51:09,298 [main] ERROR org.apache.bookkeeper.http.vertx.VertxHttpServer - Failed to start org.apache.bookkeeper.http server on port 8000
java.lang.IllegalStateException: executor not accepting a task
        at io.netty.resolver.AddressResolverGroup.getResolver(AddressResolverGroup.java:61) ~[io.netty-netty-resolver-4.1.86.Final.jar:4.1.86.Final]
        at io.vertx.core.impl.AddressResolver.resolveHostname(AddressResolver.java:82) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.impl.VertxImpl.resolveAddress(VertxImpl.java:829) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.net.impl.AsyncResolveConnectHelper.doBind(AsyncResolveConnectHelper.java:56) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:290) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:205) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.http.impl.HttpServerImpl.listen(HttpServerImpl.java:201) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at org.apache.bookkeeper.http.vertx.VertxHttpServer$2.start(VertxHttpServer.java:85) ~[org.apache.bookkeeper.http-vertx-http-server-4.14.7.jar:4.14.7]
        at io.vertx.core.AbstractVerticle.start(AbstractVerticle.java:108) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.Verticle.start(Verticle.java:66) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.impl.DeploymentManager.lambda$doDeploy$9(DeploymentManager.java:556) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.impl.ContextImpl.executeTask(ContextImpl.java:366) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.vertx.core.impl.EventLoopContext.lambda$executeAsync$0(EventLoopContext.java:38) ~[io.vertx-vertx-core-3.9.8.jar:3.9.8]
        at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:174) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:167) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[io.netty-netty-transport-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.86.Final.jar:4.1.86.Final]
        at java.lang.Thread.run(Thread.java:833) ~[?:?]
Exception in thread "main" java.lang.IllegalStateException: Can't move to started state when closed
        at org.apache.bookkeeper.common.component.Lifecycle.moveToStarted(Lifecycle.java:134)
        at org.apache.bookkeeper.common.component.AbstractLifecycleComponent.start(AbstractLifecycleComponent.java:93)
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.lambda$start$4(LifecycleComponentStack.java:144)
        at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
        at org.apache.bookkeeper.common.component.LifecycleComponentStack.start(LifecycleComponentStack.java:144)
        at org.apache.bookkeeper.common.component.ComponentStarter.startComponent(ComponentStarter.java:85)
        at org.apache.bookkeeper.server.Main.doMain(Main.java:234)
        at org.apache.bookkeeper.server.Main.main(Main.java:208)
2023-07-07T09:51:09,307 [vert.x-eventloop-thread-8] INFO  org.apache.bookkeeper.http.vertx.VertxHttpServer - HTTP server is shutdown
2023-07-07T09:51:09,311 [component-shutdown-thread] INFO  org.apache.bookkeeper.proto.BookieServer - Shutting down BookieServer

Debashish Mallick

07/07/2023, 10:23 AM

@Lari Hotari we already upgraded to pulsar 2.10.3 (which is closer to pulsar latest release) and tried in pulsar 2.10.4 . Still seeing same issue while replaying a journal file, one of bookie is going to OutofDirectMemory.

Open in Slack

Previous Next