<@U078Y59KC3B> I'm using version 3.3.2 for both th...
# ask-ai
d
@RunLLM I'm using version 3.3.2 for both the Pulsar server and the client. The client subscribes to a partitioned topic using a
topicsPattern
regex. The topic exists and messages are being received on it, but the client doesn't seem to subscribe properly. I can see that it finds the topic, but it doesn't receive any messages. However, if I explicitly specify which partitioned topic to read from, it does receive them. What could be the issue?
r
When using a
topicsPattern
regex to subscribe to a partitioned topic in Pulsar, the client should automatically discover and subscribe to all matching partitions. However, if the client finds the topic but does not receive messages, this is often due to the pattern not matching the full topic name format, or a misconfiguration in the pattern or subscription mode. For partitioned topics, the pattern must match the full topic name, including the domain (e.g.,
persistent://tenant/namespace/topic.*
). If you omit the domain or use an incorrect pattern, the consumer may not subscribe to the partitions as expected. This behavior is confirmed in PatternTopicsConsumerImplTest.java and discussed in this issue. Additionally, ensure that the broker-side pattern evaluation is enabled and that your pattern is not exceeding the
subscriptionPatternMaxLength
limit. If you use a pattern like
public/default/topic.*
instead of
persistent://public/default/topic.*
, the consumer may not receive messages. Explicitly specifying the topic works because it bypasses the pattern matching logic. Would you like more detail or example code for correct pattern usage? Sources: - PatternTopicsConsumerImplTest.java - https://github.com/apache/pulsar/issues/21236 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
d
@RunLLM yes please
r
Certainly! Here’s a step-by-step explanation and example for subscribing to a partitioned topic using a regex pattern in Pulsar 3.3.2: 1. Pattern Must Match Full Topic Name: The regex pattern should match the full topic name, including the domain. For example, if your topic is
persistent://public/default/my-topic
, your pattern should be
persistent://public/default/my-topic.*
to match all partitions (e.g.,
persistent://public/default/my-topic-partition-0
). 2. Use topicsPattern API: In Java, use the
topicsPattern
method on the consumer builder, passing a compiled regex pattern. 3. Subscription Mode: By default, the consumer will subscribe to all matching topics. If you want to restrict to only persistent or non-persistent topics, use the
subscriptionTopicsMode
method. 4. Partition Discovery: The consumer will automatically discover new partitions if the pattern matches. Example Java Code:
Copy code
import org.apache.pulsar.client.api.*;

Pattern pattern = Pattern.compile("persistent://public/default/my-topic.*");
Consumer<byte[]> consumer = pulsarClient.newConsumer()
    .topicsPattern(pattern)
    .subscriptionName("my-subscription")
    .subscriptionType(SubscriptionType.Shared)
    .subscribe();
Key Points: - If you use just
public/default/my-topic.*
or omit the domain, the consumer may not subscribe to the partitions as expected (source). - If you specify the topic directly (e.g.,
persistent://public/default/my-topic
), it works because it bypasses the pattern logic. - Make sure your pattern does not exceed the broker’s
subscriptionPatternMaxLength
(source). - The consumer will only receive messages from topics that match the pattern at the time of subscription and as new topics are created. Let me know if you need an example in another language or more troubleshooting steps. Sources: - PatternTopicsConsumerImplTest.java - https://github.com/apache/pulsar/issues/21236 - https://github.com/apache/pulsar/issues/23750 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
d
@RunLLM The topic matching seems fine, but the client not subscribing to the topic
r
If the topic regex matches but the client is not subscribing, it may be due to the pattern not including the full domain or a broker/client configuration issue. In Pulsar 3.x, the pattern must match the full topic name (e.g.,
persistent://tenant/namespace/topic.*
). If the pattern is correct and the topic is found, but the client still does not subscribe, check if the pattern length exceeds the broker's
subscriptionPatternMaxLength
(default is 50), which will prevent subscription and log an error (source). Also, ensure broker-side pattern evaluation is enabled. If all configuration is correct and the issue persists, it could be a bug or a delay in topic discovery. Try increasing the
patternAutoDiscoveryPeriod
to poll more frequently for new topics (source). Would you like troubleshooting steps or log inspection tips? Sources: - https://github.com/apache/pulsar/issues/23750 - ConsumerBuilder.java πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
d
@RunLLM yes
r
To troubleshoot, first check your client logs for errors like "Pattern longer than maximum" or "Failed to create topic list watcher," which indicate pattern or configuration issues (source). Next, verify that your pattern matches the full topic name, including the domain, and does not exceed the
subscriptionPatternMaxLength
(default 50). You can increase this limit in the broker config if needed. Also, ensure
patternAutoDiscoveryPeriod
is set to a low value (e.g., 1 second) to speed up topic discovery. If broker-side pattern evaluation is disabled, enable it in the broker config. Finally, confirm that the topic exists and is active (has producers or subscriptions), as inactive topics may not be listed for pattern subscriptions (source). Would you like a checklist or example log messages to look for? Sources: - https://github.com/apache/pulsar/issues/23750 - https://github.com/apache/pulsar/issues/19493 πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
d
@RunLLM Show me an example on how to use a consumer with topicsPattern regex when there are multiple Pulsar Brokers and therefore multiple partitioned topics
r
To consume from multiple partitioned topics across multiple Pulsar brokers using a regex, use the
topicsPattern
API with a pattern that matches all desired topics in a namespace. The Pulsar client will automatically discover all partitions and brokers for matching topics (docs). Example (Java):
Copy code
import org.apache.pulsar.client.api.*;
import java.util.regex.Pattern;

PulsarClient client = PulsarClient.builder()
    .serviceUrl("pulsar://broker1:6650,broker2:6650")
    .build();

Pattern pattern = Pattern.compile("persistent://my-tenant/my-namespace/my-topic.*");
Consumer<byte[]> consumer = client.newConsumer()
    .topicsPattern(pattern)
    .subscriptionName("my-sub")
    .subscriptionType(SubscriptionType.Shared)
    .subscribe();
This will subscribe to all partitions of all topics matching the pattern, regardless of which broker hosts them. Would you like a code example in another language? Sources: - Pulsar Consumer Multi-Topic Docs πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.
d
@RunLLM What needs to be set in the broker.conf to make it work with partitioned topics and multi topic consumers?
r
To support partitioned topics and multi-topic (pattern) consumers, ensure the following in `broker.conf`: 1. Partitioned Topics: No special config is needed for partitioned topics, but you may want to set
maxNumPartitionsPerPartitionedTopic
to control the maximum allowed partitions (source). 2. Pattern Subscriptions: Set
enableBrokerSideSubscriptionPatternEvaluation=true
to allow efficient regex/pattern subscriptions (source). Also, adjust
subscriptionPatternMaxLength
if your regex is long. Would you like the exact config lines to add? Sources: - ServiceConfiguration.java - PIP-145 PR πŸ‘ 0 button πŸ‘Ž 0 button πŸ’‘ Hint: Mention @runllm in the thread for followups.