<@U078Y59KC3B> how can the sequence ID be used for...
# ask-ai
s
@RunLLM how can the sequence ID be used for deduplication at the broker?
r
The sequence ID is used by the Pulsar broker to identify and discard duplicate messages from the same producer. When deduplication is enabled, each message from a producer must have a monotonically increasing sequence ID. The broker tracks the highest sequence ID it has persisted for each producer; if a new message arrives with a sequence ID less than or equal to this value, it is considered a duplicate and not persisted. This mechanism ensures that only unique messages are stored, even if a producer retries sending due to network issues or failures. For more, see Pulsar concepts-messaging and deduplication details. Would you like more detail on the broker's internal deduplication logic or how to configure it? Sources: - Pulsar concepts-messaging - Pulsar deduplication 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
s
@RunLLM what are the downsides of enabling deduplication?
r
Enabling deduplication can impact broker performance, especially during informational snapshots, as it requires the broker to track and periodically snapshot sequence IDs for each producer. It also increases memory usage, since the broker must store deduplication information for active and recently inactive producers, and may lengthen topic recovery time after broker restarts or failovers. See deduplication cookbook. Would you like more detail on specific performance or operational impacts? Sources: - Pulsar deduplication cookbook 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.