pulsar #general

Join Slack

Yu Wei Sung

01/19/2023, 4:43 PM

I think so.

Slackbot

01/19/2023, 8:55 PM

This message was deleted.

Slackbot

01/20/2023, 10:40 AM

This message was deleted.

Slackbot

01/24/2023, 12:47 PM

This message was deleted.

Slackbot

01/24/2023, 6:47 PM

This message was deleted.

Slackbot

01/24/2023, 11:29 PM

This message was deleted.

Murali

01/24/2023, 11:30 PM

got the same error when i ran the tests with 3 and 5replicas of bookies and brokers . horizontal scaling didn't help much. not sure if i need to change any other config

Murali

01/24/2023, 11:31 PM

Also seeing this error in broker

Copy code

Error: Too many requests to the same Bookie while reading L542724 E352 from bookie

Murali

01/24/2023, 11:33 PM

Copy code

2023-01-24T23:23:04,005+0000 [AsyncHttpClient-60-1] WARN  org.apache.pulsar.client.admin.internal.BaseResource - [<http://apache-pulsar-test10-broker-0.apache-pulsar-test10-broker.pulsar.svc.cluster.local:8080/admin/v2/namespaces/benchmark/ns-1JvIwtQ/0xb8000000_0xc0000000/split?unload=true>] Failed to perform http put request: javax.ws.rs.ClientErrorException: HTTP 412 Invalid upper boundary for bundle
2023-01-24T23:23:04,005+0000 [pulsar-modular-load-manager-30-1] ERROR org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Failed to split namespace bundle benchmark/ns-1JvIwtQ/0xb8000000_0xc0000000
org.apache.pulsar.client.admin.PulsarAdminException$PreconditionFailedException: Invalid upper boundary for bundle
	at org.apache.pulsar.client.admin.internal.BaseResource.getApiException(BaseResource.java:240) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
	at org.apache.pulsar.client.admin.internal.BaseResource$1.failed(BaseResource.java:134) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
	at org.glassfish.jersey.client.JerseyInvocation$1.failed(JerseyInvocation.java:882) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation$1.completed(JerseyInvocation.java:863) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.processResponse(ClientRuntime.java:229) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.ClientRuntime.access$200(ClientRuntime.java:62) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.lambda$response$0(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:288) ~[org.glassfish.jersey.core-jersey-common-2.34.jar:?]
	at org.glassfish.jersey.client.ClientRuntime$2.response(ClientRuntime.java:173) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$apply$1(AsyncHttpConnector.java:247) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at org.apache.pulsar.client.admin.internal.http.AsyncHttpConnector.lambda$retryOperation$4(AsyncHttpConnector.java:289) ~[org.apache.pulsar-pulsar-client-admin-original-2.10.2.jar:2.10.2]
	at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506) ~[?:?]
	at java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073) ~[?:?]
	at org.asynchttpclient.netty.NettyResponseFuture.loadContent(NettyResponseFuture.java:222) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at org.asynchttpclient.netty.NettyResponseFuture.done(NettyResponseFuture.java:257) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.finishUpdate(AsyncHttpClientHandler.java:241) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleChunk(HttpHandler.java:114) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at org.asynchttpclient.netty.handler.HttpHandler.handleRead(HttpHandler.java:143) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at org.asynchttpclient.netty.handler.AsyncHttpClientHandler.channelRead(AsyncHttpClientHandler.java:78) ~[org.asynchttpclient-async-http-client-2.12.1.jar:?]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.CombinedChannelDuplexHandler$DelegatingChannelHandlerContext.fireChannelRead(CombinedChannelDuplexHandler.java:436) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:327) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:299) ~[io.netty-netty-codec-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.CombinedChannelDuplexHandler.channelRead(CombinedChannelDuplexHandler.java:251) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:722) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:658) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:584) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:496) ~[io.netty-netty-transport-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:995) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[io.netty-netty-common-4.1.77.Final.jar:4.1.77.Final]
	at java.lang.Thread.run(Thread.java:829) ~[?:?]
Caused by: javax.ws.rs.ClientErrorException: HTTP 412 Invalid upper boundary for bundle
	at org.glassfish.jersey.client.JerseyInvocation.createExceptionForFamily(JerseyInvocation.java:985) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.convertToException(JerseyInvocation.java:967) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	at org.glassfish.jersey.client.JerseyInvocation.access$700(JerseyInvocation.java:82) ~[org.glassfish.jersey.core-jersey-client-2.34.jar:?]
	... 54 more

Slackbot

01/24/2023, 11:39 PM

This message was deleted.

Liam

01/25/2023, 4:07 AM

Hello, we updated one of our sources to have the following additional fields.

Copy code

snapshot.mode = "never"
singal.data.collection = "public.dbz_signal"

After this, our Pulsar deployment become very unstable and the broker, function & bookkeeper statefulsets went into a CrashLoopBackOff. After a few minutes, they stabilised but our existing sources are now no longer visible. When we try to create the source again, we get a 504 Gateway Timeout. Where should we look to try to identify a cause and a remediation? We have seen a lot of these errors in the broker & function statefulset. org.apache.zookeeper.KeeperException$BadVersionException: KeeperErrorCode = BadVersion for /managed-ledgers/public/functions/persistent/metadata

Slackbot

01/25/2023, 7:56 AM

This message was deleted.

Slackbot

01/25/2023, 10:53 AM

This message was deleted.

Frank Hsueh

01/25/2023, 6:53 PM

Can Pulsar support millions of ephemeral topics ? That is, persisted topics are created, live for 1d or so, then are deleted. But 1M to 10M are live at any time. The thinking is that each topic is a private event queue for a session of sorts.

David K

01/25/2023, 8:07 PM

Currently Pulsar can support ~1M topics, but it does requires some tuning of the metadata store, etc to achieve. See this PIP for more details.

thankyou 1

tomaszmichalak

01/27/2023, 11:21 AM

Hi, we plan to use Pulsar for our massive HTML page generation. We generate HTML pages with page path as a message key and markup as an event value. Our volume is around 1000k pages with a 30kB size (~30GB per all-in-one generation). Once the pages are ready, they are exposed to a single topic and read by web servers. These web servers can start on demand - there can be 0 web servers for some time. Our goals: 1. read all HTML pages (latest versions) by web servers at any point 2. keep the latest version of each page even when no web servers are connected (no subscription) 3. remove old messages (old pages) What are your recommendations around message retention settings and topic compaction?

Slackbot

01/30/2023, 9:51 AM

This message was deleted.

Slackbot

01/30/2023, 7:03 PM

This message was deleted.

Slackbot

01/31/2023, 4:18 AM

This message was deleted.

Slackbot

01/31/2023, 11:23 PM

This message was deleted.

Slackbot

01/31/2023, 11:42 PM

This message was deleted.

🙌 1

Slackbot

02/01/2023, 3:29 AM

This message was deleted.

Slackbot

02/01/2023, 6:35 AM

This message was deleted.

Slackbot

02/01/2023, 1:02 PM

This message was deleted.

Slackbot

02/01/2023, 4:14 PM

This message was deleted.

Slackbot

02/01/2023, 9:02 PM

This message was deleted.

Slackbot

02/01/2023, 10:51 PM

This message was deleted.

Slackbot

02/02/2023, 11:07 PM

This message was deleted.

👍 5

👀 3

Slackbot

02/03/2023, 7:35 AM

This message was deleted.

wangjie

02/03/2023, 2:33 PM

Hello, everyone. It’s a pleasure to join you

🙌 3