@RunLLM pulsar.dedup subscription is not moving

If the pulsar.dedup subscription is not moving, it usually means the deduplication cursor is stuck and not advancing, which can prevent ledger cleanup and cause storage issues. This can happen if deduplication is disabled but the deduplication cursor (pulsar.dedup) still exists, or if there are failures in the deduplication task or snapshot process. Restarting the broker may sometimes help, but the underlying issue is often that the deduplication cursor is not being properly advanced or deleted when deduplication is disabled. This is a known issue and has been discussed in detail, with fixes proposed to ensure the deduplication cursor is cleaned up when deduplication is turned off or fails to initialize properly. For more, see this PR discussion and the deduplication cookbook. Would you like more detail on troubleshooting or resolving this issue? Sources: - PR: Disabled deduplication, but ledger-store will not be deleted - deduplication cookbook 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@Viji i think this is the issue with dedup .

Yes, if deduplication fails and the deduplication cursor (pulsar.dedup) does not advance, it can artificially inflate the topic's backlog size. This may cause the backlog quota to be exceeded, even if other subscriptions are up-to-date, because backlog calculations consider the slowest cursor—including system cursors like pulsar.dedup. As a result, producers may be blocked or other backlog quota policies may be triggered for all subscriptions on the topic, not just the affected one. See this PR discussion and backlog quota docs. Would you like more detail on how to resolve or monitor this scenario? Sources: - PR: Producer is blocked on creation because backlog exceeded on topic, when dedup is enabled and no producer is there - backlog quota docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM "deduplicationStatus" : "Failed", means

"deduplicationStatus" : "Failed" means the broker attempted to enable or disable message deduplication for the topic, but the operation did not complete successfully due to an error (such as a cursor or ledger issue). In this state, deduplication is not functioning, and the topic may not be protected against duplicate messages. See MessageDeduplication.java for status handling. Would you like more detail on possible causes or recovery steps? Sources: - MessageDeduplication.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM yes

Possible causes for "deduplicationStatus" : "Failed" include exceptions during deduplication cursor operations (such as opening, deleting, or replaying the cursor), managed ledger errors, or snapshot failures. When this status occurs, deduplication is not active, and the topic may require manual intervention—such as unloading/reloading the topic, restarting the broker, or fixing underlying storage/cursor issues—to recover. See MessageDeduplication.java and PR 24511 for details on error handling and recovery. Would you like step-by-step troubleshooting guidance? Sources: - MessageDeduplication.java - PR: Fix deduplication replay might never complete for exceptions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to enable and disable deduplication for the topic

To enable or disable deduplication for a topic, use the Pulsar admin CLI: - Enable:

pulsar-admin topics set-deduplication persistent://tenant/namespace/topic --enable

- Disable:

pulsar-admin topics set-deduplication persistent://tenant/namespace/topic --disable

You can also use the equivalent API or StreamNative Console. See deduplication cookbook and admin reference. Would you like more detail or code examples? Sources: - deduplication cookbook - pulsar-admin reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM can i reset dedup cursor in pulsar

There is no direct command or API to reset the deduplication cursor (pulsar.dedup) in Pulsar. The deduplication cursor is managed internally by the broker and is not exposed for manual reset via admin tools. You can only enable, disable, or remove deduplication for a topic, which may delete the dedup cursor as a side effect. See MessageDeduplication.java and admin reference. Would you like more detail on deduplication cursor management or recovery options? Sources: - MessageDeduplication.java - admin reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM remove dedup

To remove the deduplication policy for a topic, use the Pulsar admin CLI:

pulsar-admin topics remove-deduplication persistent://tenant/namespace/topic

This command deletes the deduplication policy and may remove the deduplication cursor if deduplication is disabled. See pulsar-admin reference and StreamNative pulsarctl docs. Would you like more detail or examples for namespace-level removal? Sources: - pulsar-admin reference - StreamNative pulsarctl docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how to add the deduplication after remove ?

To add (re-enable) deduplication after removal, use the Pulsar admin CLI:

pulsar-admin topics set-deduplication persistent://tenant/namespace/topic --enable

This command enables deduplication for the topic again. See deduplication cookbook and pulsar-admin reference. Would you like more detail or namespace-level instructions? Sources: - deduplication cookbook - pulsar-admin reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.