@RunLLM is it possible to set retention policy for topics with a specific name pattern?

It seems likely that you cannot directly set a retention policy for all topics matching a name pattern in a single command. Retention policies in Pulsar and StreamNative are set per-topic or per-namespace, not via topic name patterns. You can set retention for a namespace (which applies to all topics in that namespace) or for individual topics by name, but not for a regex or wildcard pattern of topic names. See Pulsar retention policy docs and StreamNative migration guide. Would you like more detail on how to automate setting retention for multiple topics matching a pattern? Sources: - Pulsar retention policy docs - StreamNative migration guide 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how can I set the retention policy for a specific topic?

To set the retention policy for a specific topic, use the Pulsar admin CLI or API. For example, with the CLI:

pulsar-admin topics set-retention tenant/namespace/topic --time 100m --size 1G

This sets the retention time and size for the given topic. You must specify the full topic name and have tenant admin permissions. See StreamNative pulsarctl docs. Would you like more detail or examples for other tools or APIs? Sources: - StreamNative pulsarctl docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM is the

defaultRetentionSizeInMB

the size of data on each bookkeeper node or the total size of the messages in a topic?

My understanding is that

defaultRetentionSizeInMB

refers to the total size of messages retained for a topic, not per BookKeeper node. This quota applies to the sum of all retained messages in the topic, regardless of how they are distributed across BookKeeper nodes. See Pulsar retention policy docs and ManagedLedgerConfig.java. Would you like more detail on how this interacts with BookKeeper storage or retention enforcement? Sources: - Pulsar retention policy docs - ManagedLedgerConfig.java 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how can I configure the retention policy for backlog quota?

To configure the retention policy for backlog quota, use the

set-backlog-quota

command at the namespace or topic level. You can specify the size or time threshold and the retention policy (

producer_request_hold

,

producer_exception

, or

consumer_backlog_eviction

). For example:

pulsar-admin namespaces set-backlog-quota tenant/namespace --limit 2G --policy producer_request_hold

or for time-based quota:

pulsar-admin namespaces set-backlog-quota tenant/namespace --limitTime 3600 --policy consumer_backlog_eviction --type message_age

See Pulsar backlog quota docs for details. Would you like more detail on the meaning of each policy or how to apply this at the topic level? Sources: - Pulsar backlog quota docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM how can I configure the backlog quota retention policy for a specific topic?

To configure the backlog quota retention policy for a specific topic, use the

set-backlog-quota

command with the full topic name. For example:

pulsar-admin topics set-backlog-quota tenant/namespace/topic --limit 2G --policy producer_request_hold

You can also set a time-based quota with

--limitTime

and

--type message_age

. Valid policies are

producer_request_hold

,

producer_exception

, and

consumer_backlog_eviction

. See Pulsar admin reference. Would you like more detail or examples for other tools? Sources: - Pulsar admin reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM what is the difference between setting backlog quota for time with

consumer_backlog_eviction

and

set-message-ttl

?

It seems likely that setting a backlog quota for time with

consumer_backlog_eviction

will cause the broker to discard the oldest unacknowledged messages when the backlog age exceeds the limit, affecting only unacknowledged messages. In contrast,

set-message-ttl

sets a time-to-live for all unacknowledged messages, after which they are automatically marked as acknowledged and eligible for deletion, regardless of backlog quota. TTL is like a stopwatch per message, while backlog quota is enforced at the topic or namespace level and triggers eviction only when the quota is exceeded. See Pulsar retention and expiry docs. Would you like a step-by-step comparison or more detail on their interactions? Sources: - Pulsar retention and expiry docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@RunLLM I would like a step-by-step comparison or more detail on their interactions

My understanding is that the main differences and interactions between backlog quota with

consumer_backlog_eviction

and

set-message-ttl

are as follows: 1. Backlog Quota with `consumer_backlog_eviction`: - Applies a time (or size) limit to the backlog of unacknowledged messages for a topic or namespace. - When the oldest unacknowledged message exceeds the time limit, the broker will automatically evict (delete) the oldest messages from the backlog until the quota is no longer exceeded. - This only affects unacknowledged messages and is enforced when the quota is breached, not on a per-message timer. - Example: If you set a 1-hour backlog quota with

consumer_backlog_eviction

, any unacknowledged messages older than 1 hour will be removed when the quota check runs. - See Pulsar backlog quota docs. 2. `set-message-ttl`: - Sets a time-to-live (TTL) for all unacknowledged messages in a namespace or topic. - Each message gets a timer upon arrival; if it is not acknowledged within the TTL, it is automatically marked as acknowledged and becomes eligible for deletion. - TTL is enforced per message, regardless of backlog size or age of other messages. - Example: If you set a TTL of 1 hour, every message not acknowledged within 1 hour will be expired and deleted, even if the backlog quota is not exceeded. - See Pulsar retention and expiry docs. Interaction: If both are set, whichever policy triggers first will remove the message. TTL may expire messages before the backlog quota is reached, or backlog quota eviction may remove messages if the backlog grows too old or large before TTL expires. Sources: - Pulsar backlog quota docs - Pulsar retention and expiry docs 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.