https://pulsar.apache.org/ logo
Join Slack
Powered by
This message was deleted.
# general
s
This message was deleted.
t
and I have tried this patch in 
<https://github.com/apache/pulsar/issues/21557>
, but not worked.
d
Errors like this occur when you are trying to send a message whose size exceeds the capacity of the network frame. The max capacity of the frame is set to 5253120 bytes, which matches the max message size in Pulsar. According to the stack trace, your message is 3405774852 bytes over this limit. If you are really trying to send messages this big, you should consider enabling message chunking.
👀 1
t
Thanks for your reply, but I don’t think I have a message with so large size produced
d
Do you have batching enabled on rhe producer?
t
Yes
d
That might account for the large payload.
t
OK, I will try to add the chunking params in my client.
btw, could it be the haproxy cause this? I use haproxy in front of my broker. and i noticed that official docs say haproxy is not supported . 
The proxy in Pulsar acts as a reverse proxy, and creates a gateway in front of brokers. Proxies such as Apache Traffic Server (ATS), HAProxy, Nginx, and Envoy are not supported in Pulsar.
https://pulsar.apache.org/docs/3.1.x/concepts-proxy-sni-routing/
d
That could he an issue as well.
t
so what should i do to serve my broker in one ip ?
d
The pulsar proxy or istio are good options
t
maybe i need many pulsar proxy instance, so i need add an haproxy in front of my pulsar proxy.
Are you Chinese? you profile say you are working in StreamNaive. maybe we could communicate in Mandrin
sometimes the error is like this:
Copy code
10.50.100.197 : 2023-12-07T10:11:11,768+0800 [pulsar-io-3-45] ERROR org.apache.pulsar.broker.service.Producer - [PersistentTopic{topic=<persistent://dc/dc_binlog_erp/binlog_dws_msku_trade_1h_hot_rf}>] [mysql-10-50-40-88] Failed to verify checksum
10.50.100.197 : 2023-12-07T10:11:11,769+0800 [pulsar-io-3-45] WARN  org.apache.pulsar.broker.service.ServerCnx - [/10.50.96.135:59866] Got exception java.lang.IndexOutOfBoundsException: readerIndex(12) + length(1) exceeds writerIndex(12): PooledSlicedByteBuf(ridx: 12, widx: 12, cap: 12/12, unwrapped: PooledUnsafeDirectByteBuf(ridx: 8212, widx: 11441, cap: 16384))
10.50.100.197 : 	at io.netty.buffer.AbstractByteBuf.checkReadableBytes0(AbstractByteBuf.java:1442)
10.50.100.197 : 	at io.netty.buffer.AbstractByteBuf.readByte(AbstractByteBuf.java:730)
10.50.100.197 : 	at org.apache.pulsar.common.api.proto.LightProtoCodec.readVarInt(LightProtoCodec.java:140)
10.50.100.197 : 	at org.apache.pulsar.common.api.proto.BaseCommand.parseFrom(BaseCommand.java:2694)
10.50.100.197 : 	at org.apache.pulsar.common.protocol.PulsarDecoder.channelRead(PulsarDecoder.java:122)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
10.50.100.197 : 	at io.netty.handler.flow.FlowControlHandler.dequeue(FlowControlHandler.java:202)
10.50.100.197 : 	at io.netty.handler.flow.FlowControlHandler.channelRead(FlowControlHandler.java:164)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
10.50.100.197 : 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346)
10.50.100.197 : 	at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:333)
10.50.100.197 : 	at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:454)
10.50.100.197 : 	at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
10.50.100.197 : 	at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
10.50.100.197 : 	at org.apache.pulsar.common.protocol.OptionalProxyProtocolDecoder.channelRead(OptionalProxyProtocolDecoder.java:52)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
10.50.100.197 : 	at io.netty.handler.flush.FlushConsolidationHandler.channelRead(FlushConsolidationHandler.java:152)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412)
10.50.100.197 : 	at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440)
10.50.100.197 : 	at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420)
10.50.100.197 : 	at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
10.50.100.197 : 	at io.netty.channel.epoll.AbstractEpollStreamChannel$EpollStreamUnsafe.epollInReady(AbstractEpollStreamChannel.java:800)
10.50.100.197 : 	at io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:499)
10.50.100.197 : 	at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397)
10.50.100.197 : 	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
10.50.100.197 : 	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
10.50.100.197 : 	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
10.50.100.197 : 	at java.base/java.lang.Thread.run(Thread.java:842)
8 Views
Open in Slack
PreviousNext
Powered by