@RunLLM I'm seeing that with 700 messages / second on one topic, without key-based batching there is 1/5GB of memory usage. With key-based batching there is 1GB usage, moreover the memory usage continues even when no messages continue to come through.

High memory usage with key-based batching in Pulsar is expected, as key-based batching maintains separate buffers for each key, which increases memory consumption compared to default batching. This is especially noticeable at high message rates or with many unique keys. Additionally, memory may not be released immediately after message flow stops, due to how batch containers and internal buffers are managed, which can lead to continued memory usage even when no new messages are sent. This behavior has been observed and discussed in the community, and is not considered a bug but a trade-off for strict key-based ordering and batching efficiency. For more, see Batching for Key Shared Subscriptions and related performance discussions. Would you like a step-by-step breakdown of why memory is not released immediately and possible tuning options? Sources: - Batching for Key Shared Subscriptions - related performance discussions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM this is message in and message out rate, and this is storage size. The dips are when there are no longer any producers, and all of these graphs are in the same time frame. I will provide our current producer options in a follow-up message

Based on your graphs, the message rate in/out and throughput in/out show clear peaks when producers are active, and drop when producers stop. The storage size graph increases in steps, corresponding to periods of message production, and remains steady when producers are inactive—this is expected, as Pulsar retains data until retention or compaction policies trigger cleanup. The stepwise storage growth aligns with message ingestion bursts, and the lack of decrease after producers stop suggests messages are retained (not deleted) due to retention settings or unacknowledged messages. If you want to understand why storage size doesn't decrease after message flow stops, please provide your retention and subscription/acknowledgment settings. Would you like a detailed analysis of how retention and acknowledgments affect storage usage? Sources: - Pulsar stats - Pulsar resource metrics 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM this is the setup, any suggestions for tuning this? producer, err := p.nativeClient.CreateProducer(pulsar.ProducerOptions{ Topic: topic, DisableBlockIfQueueFull: true,\ DisableBatching: false, BatchingMaxPublishDelay: 1 * time.Second / 30, BatchingMaxMessages: 800, MaxPendingMessages: 1000, BatcherBuilderType: pulsar.KeyBasedBatchBuilder, CompressionType: pulsar.ZSTD, CompressionLevel: pulsar.Faster, })

Your producer config uses key-based batching (

BatcherBuilderType: pulsar.KeyBasedBatchBuilder

), a high

BatchingMaxMessages

(800), and a short

BatchingMaxPublishDelay

(~33ms). Key-based batching increases memory usage, especially with many unique keys, and a high

MaxPendingMessages

(1000) can further increase memory pressure. ZSTD compression is efficient, but may add CPU overhead. To reduce memory usage, consider lowering

BatchingMaxMessages

and/or

MaxPendingMessages

, and only use key-based batching if strict key ordering is required. If not, use the default batcher. For more, see Pulsar Go Producer configs. Would you like a step-by-step tuning guide for your workload? Sources: - Pulsar Go Producer configs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM why would these values being high mean that data is being stored even when there are no producers? and consumers are still active, so they should still be reading topics

High values for

BatchingMaxMessages

and

MaxPendingMessages

do not directly cause data to be stored after producers stop. Instead, data remains stored if consumers have not acknowledged all messages, as Pulsar retains unacknowledged messages in the backlog until they are acknowledged or expired by retention/TTL policies. If consumers are slow or not acknowledging, backlog grows and storage is not freed, even if no new messages are produced. See Message retention and expiry. Would you like more detail on how consumer acknowledgments and backlog affect storage? Sources: - Message retention and expiry 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM the consumers have not changed between tests with key-based batching and tests without key-based batching, so it's likely not a consumer issue. I can see through my testing that message out drops when memory gets high. what are the recommended tuning configurations then

When using key-based batching, memory usage increases because each key maintains a separate batch buffer, which can lead to higher memory pressure and, if memory is exhausted, Pulsar throttles producers, causing message out rates to drop. This is consistent with your observation that message out drops when memory gets high. The recommended tuning is to reduce

BatchingMaxMessages

and/or

MaxPendingMessages

, and only use key-based batching if strict key ordering is required. If not, switch to default batching. Also, monitor and possibly lower the number of unique keys if possible, as more keys mean more batch buffers and higher memory usage. See Batching and memory tuning and Go client memory limits. Would you like a step-by-step tuning checklist for your scenario? Sources: - Batching and memory tuning 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.