This message was deleted.

Yes, you can use chunked messages to send larger messages

Thank you. What about memory footprint on the producer side? The client SDK send accepts a byte array. Can I stream data to a OutputStream? I did not find that in the SDK. Or one solution would be to use a great number of small chunks?

The Pulsar client buffers all of the chunks in an internal buffer before reconstituting the message back together. Therefore, you will need to have enough free memory to hold the entire contents of the chunked message

Thank you. So when I need to send messages with a large payload, this payload will have to fit into producer and consumer memory, even if the message is chunked. Then what is the point of chunking messages? What if I just send not-chunked messages with a large payload?

All messaging systems have a maximum message size (which is configurable), For Kafka it is 1Kb, for Pulsar it is 5 Mb. Either way, when you try to publish a message larger than the maximum size, it will get rejected. Chunked messages allow you to bypass this restriction by sending a sequence of 5 MB chunks.

Ok. The max message size is here for a reason. So if I increase the maximum message size setting on the broker side, then what should I expect? What are the drawbacks? I should consider tuning the memory settings on the broker / bookies nodes? Anything else?

Memory settings and networking settings on the broker will need to be adjusted upward to accommodate the larger messages. Larger memory settings on the producers and consumers (who have to buffer these larger messages). is a nice discussion of the rational for message chunking. How big of a message size are you considering?

100Mb

consistently (80+% of the messages) or just occasionally? If the former, then you should adjust the max message size and tune your brokers, if the later then chunked messages would be best.

Thank you. For message chunking, I need to consume in the right order to aggregate them. It means no shared subscription. Can I use chunking with a key shared subscription? It’s an ordered one

Correct, just failover and exclusive are supported. With the Key-shared subscription, it is possible for key assignments to move from one consumer to another under certain conditions.

It would be great if pulsar could understand that messages are chunks of the same original message and so deliver them to the same consumer using shared or key-shared subscription, for messaging use-cases.