This message was deleted.
# generals
Slackbot
07/16/2023, 10:42 AMThis message was deleted.
a
Asaf Mesika
07/16/2023, 2:54 PM@Penghui Any idea?
y
Yu Wei Sung
07/16/2023, 11:25 PMsearch jute.maxbuffer in https://zookeeper.apache.org/doc/r3.8.0/zookeeperAdmin.html
The explanation is under unsafe options
Yu Wei Sung
07/16/2023, 11:32 PMpacket len 12644126 ~ 10MB. this means your znode size is bigger than 10MB. As a pulsar user, you need to increase jute.maxbuffer. as a pulsar developer, you can try search which znode is over the limit and try to find how broker/bookkeeper store info in this znode and reduce the content size.
m
Moty
07/17/2023, 7:20 AMwhat does it mean that my znode is bigger than 10MB?
I’m deploying Pulsar on top of EKS and i’m giving zookeeper nodes a 20Gi EBS volume for each node, so 10MB shouldn’t be any problem.
do you mean: a message size that zookeeper is handling?
Moty
07/17/2023, 7:36 AMwhat does this line mean?
When jute.maxbuffer in the client side is less than the server side, the client wants to read the data exceeds jute.maxbuffer in the client side, the client side will get java.io.IOException: Unreasonable length or Packet len is out of range!
who is the client in my case? pulsar broker?
Moty
07/17/2023, 7:36 AMalso, I see it’s not recommended to change that in production
p
Penghui
07/17/2023, 7:56 AMThe client is pulsar-broker, which use 10MB by default https://github.com/apache/pulsar/blob/ecd16d68e29700cc9521e6d77d124a403c3486c9/bin/pulsar#L288, you can try to change Pulsar Env https://github.com/apache/pulsar/blob/ecd16d68e29700cc9521e6d77d124a403c3486c9/conf/pulsar_env.sh#L86 to increase the jute.maxbuffer size.
a
Asaf Mesika
07/17/2023, 8:03 AM@Moty, Zookeeper is a tree-like system (like a linux file system, with directories and files). Like any tree in computer science, it is built of nodes, referred to as ZNode (Zookeeeper node). Each such node can hold data (an array of bytes) and list of child nodes.
What does Jute.maxBuffer means? ChatGPT 4 (3.5 gives really bad result) to the rescue:
“jute.maxbuffer” is a configuration property in Apache ZooKeeper that specifies the maximum size of the data that ZooKeeper will accept from a client for processing. It’s particularly relevant when working with larger amounts of data.
This property is part of ZooKeeper’s underlying data serialization library, Jute. By default, the max buffer size is typically set to 1 megabyte (1MB), which means any attempt to write data larger than this limit to ZooKeeper will result in an exception.Pulsar holds a ZK client since Pulsar stores information in ZK, like the list of ledger IDs per topic.
Asaf Mesika
07/17/2023, 8:04 AM@Penghui Should Pulsar add to the error the node that was maxed out in size?
p
Penghui
07/17/2023, 8:08 AM@Asaf Mesika Yes, that makes sense.
a
Asaf Mesika
07/17/2023, 8:32 AM@Moty If you have the stacktrace, would like to contribute a fix to this?
m
Moty
07/17/2023, 2:17 PM@Asaf Mesika I don’t have the stacktrace and the change you are talking about is a change in pulsar code itself, not in the chart as I understand
Regarding my issue, do you think I should increase the Jute.maxBuffer for pulsar borker?
a
Asaf Mesika
07/17/2023, 3:08 PM@Moty Yes, it’s in the Pulsar code.
Based on what was written here, yes, increase it.
y
Yu Wei Sung
07/17/2023, 3:20 PMjute.maxBuffer on server and client MUST match. make sure you update zk, broker and bookie (both are zk clients).