Dan Ger
05/23/2025, 2:02 PMtopicsPattern regex. The topic exists and messages are being received on it, but the client doesn't seem to subscribe properly. I can see that it finds the topic, but it doesn't receive any messages. However, if I explicitly specify which partitioned topic to read from, it does receive them. What could be the issue?RunLLM
05/23/2025, 2:02 PMtopicsPattern regex to subscribe to a partitioned topic in Pulsar, the client should automatically discover and subscribe to all matching partitions. However, if the client finds the topic but does not receive messages, this is often due to the pattern not matching the full topic name format, or a misconfiguration in the pattern or subscription mode. For partitioned topics, the pattern must match the full topic name, including the domain (e.g., persistent://tenant/namespace/topic.*). If you omit the domain or use an incorrect pattern, the consumer may not subscribe to the partitions as expected. This behavior is confirmed in PatternTopicsConsumerImplTest.java and discussed in this issue.
Additionally, ensure that the broker-side pattern evaluation is enabled and that your pattern is not exceeding the subscriptionPatternMaxLength limit. If you use a pattern like public/default/topic.* instead of persistent://public/default/topic.*, the consumer may not receive messages. Explicitly specifying the topic works because it bypasses the pattern matching logic.
Would you like more detail or example code for correct pattern usage?
Sources:
- PatternTopicsConsumerImplTest.java
- https://github.com/apache/pulsar/issues/21236 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.Dan Ger
05/23/2025, 2:11 PMRunLLM
05/23/2025, 2:11 PMpersistent://public/default/my-topic, your pattern should be persistent://public/default/my-topic.* to match all partitions (e.g., persistent://public/default/my-topic-partition-0).
2. Use topicsPattern API: In Java, use the topicsPattern method on the consumer builder, passing a compiled regex pattern.
3. Subscription Mode: By default, the consumer will subscribe to all matching topics. If you want to restrict to only persistent or non-persistent topics, use the subscriptionTopicsMode method.
4. Partition Discovery: The consumer will automatically discover new partitions if the pattern matches.
Example Java Code:
import org.apache.pulsar.client.api.*;
Pattern pattern = Pattern.compile("persistent://public/default/my-topic.*");
Consumer<byte[]> consumer = pulsarClient.newConsumer()
.topicsPattern(pattern)
.subscriptionName("my-subscription")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
Key Points:
- If you use just public/default/my-topic.* or omit the domain, the consumer may not subscribe to the partitions as expected (source).
- If you specify the topic directly (e.g., persistent://public/default/my-topic), it works because it bypasses the pattern logic.
- Make sure your pattern does not exceed the brokerβs subscriptionPatternMaxLength (source).
- The consumer will only receive messages from topics that match the pattern at the time of subscription and as new topics are created.
Let me know if you need an example in another language or more troubleshooting steps.
Sources:
- PatternTopicsConsumerImplTest.java
- https://github.com/apache/pulsar/issues/21236
- https://github.com/apache/pulsar/issues/23750 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.Dan Ger
05/26/2025, 10:21 AMRunLLM
05/26/2025, 10:21 AMpersistent://tenant/namespace/topic.*). If the pattern is correct and the topic is found, but the client still does not subscribe, check if the pattern length exceeds the broker's subscriptionPatternMaxLength (default is 50), which will prevent subscription and log an error (source). Also, ensure broker-side pattern evaluation is enabled.
If all configuration is correct and the issue persists, it could be a bug or a delay in topic discovery. Try increasing the patternAutoDiscoveryPeriod to poll more frequently for new topics (source).
Would you like troubleshooting steps or log inspection tips?
Sources:
- https://github.com/apache/pulsar/issues/23750
- ConsumerBuilder.java π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.Dan Ger
05/26/2025, 11:06 AMRunLLM
05/26/2025, 11:06 AMsubscriptionPatternMaxLength (default 50). You can increase this limit in the broker config if needed.
Also, ensure patternAutoDiscoveryPeriod is set to a low value (e.g., 1 second) to speed up topic discovery. If broker-side pattern evaluation is disabled, enable it in the broker config. Finally, confirm that the topic exists and is active (has producers or subscriptions), as inactive topics may not be listed for pattern subscriptions (source).
Would you like a checklist or example log messages to look for?
Sources:
- https://github.com/apache/pulsar/issues/23750
- https://github.com/apache/pulsar/issues/19493 π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.Dan Ger
05/27/2025, 11:51 AMRunLLM
05/27/2025, 11:51 AMtopicsPattern API with a pattern that matches all desired topics in a namespace. The Pulsar client will automatically discover all partitions and brokers for matching topics (docs).
Example (Java):
import org.apache.pulsar.client.api.*;
import java.util.regex.Pattern;
PulsarClient client = PulsarClient.builder()
.serviceUrl("pulsar://broker1:6650,broker2:6650")
.build();
Pattern pattern = Pattern.compile("persistent://my-tenant/my-namespace/my-topic.*");
Consumer<byte[]> consumer = client.newConsumer()
.topicsPattern(pattern)
.subscriptionName("my-sub")
.subscriptionType(SubscriptionType.Shared)
.subscribe();
This will subscribe to all partitions of all topics matching the pattern, regardless of which broker hosts them.
Would you like a code example in another language?
Sources:
- Pulsar Consumer Multi-Topic Docs π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.Dan Ger
05/27/2025, 12:03 PMRunLLM
05/27/2025, 12:03 PMmaxNumPartitionsPerPartitionedTopic to control the maximum allowed partitions (source).
2. Pattern Subscriptions: Set enableBrokerSideSubscriptionPatternEvaluation=true to allow efficient regex/pattern subscriptions (source). Also, adjust subscriptionPatternMaxLength if your regex is long.
Would you like the exact config lines to add?
Sources:
- ServiceConfiguration.java
- PIP-145 PR π 0 button π 0 button π‘ Hint: Mention @runllm in the thread for followups.