REMINDER: CFP is open for Pulsar Virtual Summit Europe, which will take place on May 23, 2023! Share your Apache Pulsar use case, tech deep dive, or best practices with the community. Submit your idea: https://hubs.ly/Q01C6Htf0 Slack Conversation

This message was deleted.

This message was deleted.

This message was deleted.

Hi, I have a single cluster and the configuration store in the cluster metadata and broker config both points to local zookeeper. Now I need to update them to point to a global zookeeper cluster. I know how to update broker config and roll restart brokers, but what about cluster metadata? How can I do that?

This message was deleted.

This message was deleted.

👋 Hi everyone! Nice to meet you all.

I have a question around cursors. A Cursor is a subscription position for a consumer. I stumbled upon the Admin API that allows to reset a cursor for a particular topic and subscription.

bin/pulsar-admin topics reset-cursor
The following option is required: [-s | --subscription]
 
Reset position for subscription to a position that is closest to timestamp or messageId.
Usage: reset-cursor [options] <persistent://tenant/namespace/topic>
  Options:
    -e, --exclude-reset-position
      Exclude the reset position, start consume messages from the next 
      position. 
      Default: false
    --messageId, -m
      messageId to reset back to ('latest', 'earliest', or 'ledgerId:entryId')
  * -s, --subscription
      Subscription to reset position on
    --time, -t
      time in minutes to reset back to (or minutes, hours, days, weeks eg: 
      100m, 3h, 2d, 5w)

You specify the topic and the subscription, but not the consumer. But as the Cursor is a subscription position for a consumer, ... how does this work? 🤔

Every subscription is shared by one or more consumers. All of the consumers on the subscription will resume consumption from the reset position.

The subscription is used on the broker side to determine where to read the messages from and dispatch to the next consumer on the subscription.

I'm particularly thinking about a key shared subscription. If I have two consumers A and B, they would get different messages based on the key of the messages. So I would have thought that each consumer would have its own cursor. But it sounds like the subscription has just one shared cursor. Is that accurate? The statement "A Cursor is a subscription position for a consumer." is then not correct I guess.

btw I asked the question also in Github discussions so that the answer can be archived for others as well: https://github.com/apache/pulsar/discussions/19634

For the Key-shared subscription there is a different mechanism on the Broker side that tracks the assignment of the consumers on the subscription to a set of keys, e.g. a key-range. The cursor keeps track of the overall subscription position, while these key-ranges are used to route the message to the corresponding consumers based on their assigned key-range(s).

In the example with consumers A and B, lets say consumer A gets key range A-J, consumer B gets key range K-Z assigned. Now they will process messages in parallel, but each consumer processes the messages in its key range sequentially. They can be at different positions in the topic as they work concurrently (especially if one consumer is faster than the other or if more messages come with keys in one of the two key ranges). If there is a single cursor per subscription, what will it track in this case? What is the "overall subscription position"? Sorry if this is a dumb question, but it hasn't really clicked for me yet.

For a non-partitioned topic, the cursor reset is the same as any other subscription type. It just moves the cursor to the message closest to the timestamp or messageId provided. Then as the Broker dispatches the messages it reads through the topic and “routes” the messages to the consumer based on the key.

For a partitioned topic, cursor management is bit more complicated, because we really need to manage multiple cursors inside a single subscription. This is all handled transparently from the consumers, but essentially there is one cursor per partition. When a reset occurs then all of these cursors are independently moved to the message closest to the timestamp or messageId provided, and consumption resumes from there.

Thanks David. What confused me is that the consumers in a key shared subscription could be at different positions. I now read more about how cursors work, and about acknowledgment holes. Now it makes sense to me because of "when acknowledgment holes exist, the cursor [...] stays before the oldest unacknowledged message.". (https://dzone.com/articles/understanding-cursors-tracking-mechanism-in-pulsar) That's why a single cursor is sufficient for a subscription.

Glad to help. I should probably update the Github discussion thread as well

Glad to help. I should probably update the Github discussion thread as well

I did update it and added the info about the acknowledgement holes and cursor behavior.

I get an error message with Pulsar after an unclean shutdown,

Message: Error while reading ledger -  ledger=93 - operation=Failed to read entry - entry=0

What can I do to recover from this?

This message was deleted.

Can you share the source code for the io.debezium.connector.yugabytedb.YugabyteDBConnector class? From the error code, it looks like it doesn’t implement the correct Java interface

This message was deleted.

This message was deleted.

This message was deleted.

This message was deleted.

The CFP for Pulsar Virtual Summit Europe 2023 closes this Friday, March 3rd! Submit here: https://sessionize.com/pulsar-virtual-summit-europe-2023/

This message was deleted.

This message was deleted.