I have a question. Pulsar 3.1.2 + Pulsar Client 3....
# generalt
Tom Korn
03/27/2024, 10:30 AMI have a question. Pulsar 3.1.2 + Pulsar Client 3.1.2
Following szenario :
I have a pulsar which have a lot of ledgers which are in "closing" status, and should finally got offloaded to gcp-buckets.
My java consumer is quiet slow and I think that it only consumes the newest entries without taking the old on from the old ledgers. It's configured with '.subscriptionInitialPosition(SubscriptionInitialPosition.Latest)'
What do exactly '.subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)', the earliest setting? Do it also read offloaded messages from bucket again? Or does it "only" start with the oldest ledger at first?
Or does it make sense to have 2 consumer? one earliest and one latest?
Tom Korn
03/27/2024, 5:54 PMSeems to be a broker problem. Even when I connect to the same full subscription with a python client... it stucks.
d
David K
03/27/2024, 6:54 PMIn short,
subscriptionInitialPosition(SubscriptionInitialPosition.Latest)'subscriptionInitialPosition(SubscriptionInitialPosition.Earliest)t
Tom Korn
03/27/2024, 6:58 PMFrom my understand this only applies to Reader-Interface (what ever it is)
and the standard consumer (like in java pulsarClient.newConsumer()...) isn't it needed
But in the doc isn't any point to the offloaded ones.
https://pulsar.apache.org/docs/3.2.x/concepts-clients/#reader
d
David K
03/27/2024, 7:02 PMYou can specify the starting position of consumption for both subscriptions and readers. In either case, ledgers that were offloaded to tiered-storage should automatically be “read”. If this is not what you are seeing, then it might be an bug.
t
Tom Korn
03/27/2024, 7:18 PMsorry for asking stupid questions, but I'm quiet new to pulsar.
When I'm right, only ledgers where all messages are ACK gets offloaded?
Right now I have the situation that the subsciption consumer stuck somewhere in an old ledger and this ledger and all new are marked for offloading, but get not offloaded. I think because the consumer is still "in it" and the other messages are not ACK'ed because it stucks somewhere somehow.
Earlier this day I've connected consumers with Earliest and Latest. Both were working on messages from the same date (I've taken the published date from the message itself). I have not seen any offloaded older messages. So that was my suggestion that it only reads the oldes (earlies) once from the subscription on topic. Not offloaded stuff.
What is the definition of "new" messages and "oldest" messages. Only un-ACK'ed messages?
When I have a durable subscription and stop the consumer. The messages will be put in the sub and stay there until I restart my consumer.
With "latest": would it only read messages that arrive after the connect time? And ignore all other, unACK'ed messages?
d
David K
03/27/2024, 7:24 PMIf you have a consumer/subscription “stuck” on a ledger that has been closed, then offloading would not occur.
David K
03/27/2024, 7:26 PM“What is the definition of “new” messages and “oldest” messages.” Pulsar has the concept of data retention where you specify how long to keep messages in a topic around. The “earliest” message represents the oldest message in the topic that has been retained, while the latest represents the most recently published message to the topic.
David K
03/27/2024, 7:28 PMA subscriber can join a topic at any point in the topic’s lifecycle, and you can use the
subscriptionInitialPositionLatestDavid K
03/27/2024, 7:29 PMAs a consumer ack’s messages, an internal cursor position is updated for the subscription, which allows a disconnected consumer to re-connect and resume processing from where it left off.
David K
03/27/2024, 7:33 PMMessages from the subscription cursor’s position forward are automatically retained, regardless of the retention policy on the topic. Thus, if you had a subscription on the topic and forgot to delete it (or have auto-deletion of subscriptions turned off), then these ledgers can be retained longer than you anticipate. See the docs for more details.
t
Tom Korn
03/27/2024, 7:36 PMokay, thanks for clarification! Will read through it
d
David K
03/27/2024, 7:38 PMAlso, look at the topic stats to get a better understanding of where your subscription cursors are in the consumption process.
David K
03/27/2024, 7:40 PMInternal stats will show the ledger and cursor status. HTH