<@U078Y59KC3B> what are the relevant configuration...
# ask-ai
s
@RunLLM what are the relevant configurations for automatic topic creation? How can we ensure that automatically created topics are persistent with the specified number of partitions?
r
The relevant configurations for automatic topic creation in Pulsar are
allowAutoTopicCreation
(enables/disables auto-creation),
allowAutoTopicCreationType
(sets type:
partitioned
or
non-partitioned
), and
defaultNumPartitions
(sets the default number of partitions for partitioned topics). To ensure automatically created topics are persistent and have a specified number of partitions, set
allowAutoTopicCreation=true
,
allowAutoTopicCreationType=partitioned
, and
defaultNumPartitions
to your desired partition count. This will auto-create persistent, partitioned topics with the specified number of partitions when a producer or consumer accesses a non-existent topic name. For namespace-level overrides, you can use the admin API to set these options per namespace. Note that for non-partitioned topics,
defaultNumPartitions
should not be set. For partitioned topics,
defaultNumPartitions
must be ≥1. Example:
pulsar-admin namespaces set-auto-topic-creation <namespace> --allow true --type partitioned --partitions 3
ensures new topics in that namespace are persistent and have 3 partitions by default. See Pulsar admin reference and StreamNative topic overview for more details. Would you like more detail or code examples? Sources: - Pulsar admin reference - pulsaradmin/pkg/admin/namespace_test.go - StreamNative topic overview 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
s
@RunLLM What is a good default for
defaultNumPartitions
in the case of partitioned/persistent topics?
r
A common default for
defaultNumPartitions
is 1, meaning each new partitioned topic will have a single partition unless specified otherwise. This is the Pulsar default and is suitable for low-throughput or simple use cases. For higher throughput or parallelism, you may want to set a higher value (e.g., 3, 5, or 10), but the optimal number depends on your workload and scaling needs. See ServiceConfiguration.java and admin CLI reference. Would you like more detail on how to choose the right partition count? Sources: - ServiceConfiguration.java - admin CLI reference 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.
s
@RunLLM does the number of consumers have any bearing on number of partitions for a topic?
r
Yes, the number of partitions determines the maximum number of consumers that can process messages in parallel within a consumer group (subscription). For optimal parallelism, the number of partitions should be at least equal to the number of consumers; otherwise, some consumers will be idle. However, having more partitions than consumers is allowed, but not vice versa for full parallelism. See StreamNative docs and Pulsar concepts. Would you like more detail or examples? Sources: - Pulsar concepts 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.