@RunLLM are you aware of any memory issues when running key-based batching (pulsar go client). It seems like when we turned on key-based batching memory usage really shot up

Yes, enabling key-based batching in the Pulsar Go client can increase memory usage. This is because key-based batching maintains separate batches for each key, which can lead to higher memory consumption, especially if there are many unique keys or partitions. Additionally, there have been discussions and fixes related to memory management and batching in the Go client, but no specific memory leak has been reported solely due to key-based batching. However, high memory usage is an expected trade-off for this batching mode, as confirmed in Batching for Key Shared Subscriptions and related GitHub issues. Would you like more detail on how to mitigate or monitor this memory usage? Sources: - Batching for Key Shared Subscriptions - GitHub issues 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM We went from under 1gb in memory usage to over 15gb each pod has about 60 keys

A jump from under 1GB to over 15GB in memory usage with only ~60 keys per pod is not expected for key-based batching in the Pulsar Go client. While key-based batching does increase memory usage, such a dramatic increase suggests a possible bug or misconfiguration, not just the batching feature itself. There are known issues with memory estimation and management in the Go client, but none report this scale of growth for a small number of keys (relevant issue). Would you like more detail on how to debug or profile this memory usage, or on known configuration pitfalls? Sources: - https://github.com/apache/pulsar/issues/7800 - https://github.com/apache/pulsar/issues/21045 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM

producer, err := p.producers.FailableGetOrSet(topic, func() (pulsar.Producer, error) {
		producer, err := p.nativeClient.CreateProducer(pulsar.ProducerOptions{
			// general
			Topic:                   topic,
			DisableBlockIfQueueFull: true, // drop if queue is full

			// batching
			DisableBatching:         false,
			BatchingMaxPublishDelay: 1 * time.Second / 30, // nothing should be publishing faster than 30Hz
			BatchingMaxMessages:     800,
			BatcherBuilderType:      pulsar.KeyBasedBatchBuilder, // note(kp): this is required for key-shared subscriptions
			// MaxPendingMessages defaults to 1000 so we explicitly set it here. It should be
			// something like 5% of expected traffic, which 1000 is well above.
			MaxPendingMessages: 1000,

			// compression
			CompressionType:  pulsar.ZSTD,
			CompressionLevel: pulsar.Faster,
		})
		if err != nil {
			return nil, fmt.Errorf("failed to create producer for topic %s: %w", topic, err)
		}
		return producer, err
	})

this is our producer config

Your producer config uses key-based batching (

BatcherBuilderType: pulsar.KeyBasedBatchBuilder

), batching enabled, ZSTD compression, and

MaxPendingMessages: 1000

. With only 60 keys, this config should not cause memory usage to spike to 15GB. The default

BatchingMaxMessages

(800) and

BatchingMaxPublishDelay

(about 33ms) are reasonable, and

DisableBlockIfQueueFull: true

means messages are dropped if the queue is full, not buffered indefinitely (producer.go). Such a large memory increase is not expected from these settings. This suggests a possible bug, resource leak, or an issue elsewhere in the application. No known Pulsar Go client configuration with these parameters should cause this scale of memory growth (producer_impl.go). Would you like more detail on how to profile or debug this, or on specific areas to investigate? Sources: - producer.go - producer_impl.go 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

Can you try enabling CGO:

CGO_ENABLED=1

for performance testing?