Suhas Karanth
03/28/2025, 8:48 AMJavaStringHash and Murmur3_32Hash for the HashingScheme in ProducerOptions. Is the choice important in any way?RunLLM
03/28/2025, 8:48 AMJavaStringHash and Murmur3_32Hash for the HashingScheme in ProducerOptions can impact the consistency of message distribution across partitions, especially when using partitioned topics. JavaStringHash is the default and uses Java's String.hashCode(), which may not be consistent across different language clients. Murmur3_32Hash, on the other hand, is recommended when producers are from different language clients, as it provides a more consistent hashing across platforms.
If your application involves multiple language clients or requires consistent partitioning across different environments, Murmur3_32Hash is generally preferred. Otherwise, if your application is Java-centric and does not require cross-language consistency, JavaStringHash can be sufficient. Would you like more detail on how these hashing schemes work or their specific use cases?
Sources:
- Apache Pulsar Concepts - Hashing Scheme
- ProducerBuilder.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.Suhas Karanth
03/28/2025, 9:06 AMBatcherBuilderType in ProducerOptions?RunLLM
03/28/2025, 9:06 AMBatcherBuilderType field in ProducerOptions specifies the type of batch builder to use when batching is enabled for a producer. It determines how messages are grouped into batches before being sent to the broker. The available options are DefaultBatchBuilder and KeyBasedBatchBuilder.
DefaultBatchBuilder batches messages without considering their keys, which is suitable for general use cases. KeyBasedBatchBuilder, on the other hand, ensures that messages with the same key are batched together, which is particularly important for maintaining message order in scenarios like Key_Shared subscriptions. This choice is crucial when message ordering by key is required, as it affects how messages are dispatched to consumers. Would you like more detail on how these batch builders operate or their specific use cases?
Sources:
- ProducerBuilder.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.Suhas Karanth
03/28/2025, 9:14 AMKeyBasedBatchBuilder have a negative performance implication if the messages do not have an ordering key associated?RunLLM
03/28/2025, 9:14 AMKeyBasedBatchBuilder without an ordering key can lead to inefficiencies. This batch builder is designed to group messages with the same key into the same batch, ensuring order preservation for Key_Shared subscriptions. If messages lack an ordering key, they may be batched less efficiently, potentially resulting in smaller batch sizes and increased network overhead due to more frequent sends.
If your use case does not require key-based ordering, it might be more efficient to use the DefaultBatchBuilder, which does not consider keys and can batch messages more effectively, maximizing throughput. Would you like more detail on optimizing performance with different batch builders?
Sources:
- https://github.com/apache/pulsar-client-go/pull/1252
- https://pulsar.apache.org/docs/3.2.x/client-libraries-consumers#key_shared 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.