Hi,
Seems to me an answer to my question could be...
# generala
Artur Jablonski
09/30/2025, 6:24 AMHi,
Seems to me an answer to my question could be quite useful to a lot of people new to Pulsar that try to design around it with constraint of effectively one delivery semantics.
Is there anything I can do to get some attention to it from Pulsar community?
a
Asaf Mesika
10/01/2025, 9:18 AMHey man
Asaf Mesika
10/01/2025, 9:18 AMI couldn’t figure out option 2
Asaf Mesika
10/01/2025, 9:19 AMOption 1 looks right. Not sure I know anyone using it such that you protect against producer restart
a
Artur Jablonski
10/20/2025, 7:32 AMHey Asaf. Thanks for your response.
Well if you're after effectively once delivery guarantee then you should also consider the producer crashing, right? Then if your data source is such that the sequence number is somehow permanently associated with data items like in example of flat text file where line number is fixed forever to the same well line in text file (assuming immutable file) or offset in Kafka topic where an offset if permanently and forever fixed to the same item in Kafka log, then you're good.
However if your source cant have such association, like say JMS queue, where items are unordered, then you can only rely on Producer keeping some internal counter, that counter. And as long as Producer is running this works, but if Producer crashes together with the counter, then it's possible duplicate will be delivered on the Pulsar topic.
So if you're designing an application that communicates over Pulsar and you ingest data from that kind of JMS queue, then you still need to take into consideration of that possibility and need some form of deduplication on the consumer side.
I am just trying to see if this is really the case or am I missing sth.
Artur Jablonski
10/20/2025, 7:39 AMas for "Option 2"
Let's go back to that JMS queue, but this time let's imagine that thi s JMS implementation supports some form of 2phase commit. Since Pulsar also can participate in distributed transaction (like shown in the article about Flink Sink I linked in the discussion), then if I can make:
1. Consuming from JMS queue
2. Publishing to Pulsar topic
3. Acknowledging JMS message
transactionaly bound in all-or-nothing semantics, then I can still achieve Exactly once semantics even if the data source in unordered and even if there's no sequence number that can be associated with data items like in "Option 1"
Artur Jablonski
10/20/2025, 7:44 AMAnd if all that is true and since the particular JMS broker I work with does not support some transactional API, then the conclusion is that you CAN'T have effectively once semantics when ingesting data from JMS queue to pulsar topic.
I wanted to run this by Pulsar community to double check that this reasoning makes sense.
a
Asaf Mesika
10/20/2025, 1:33 PMI am just trying to see if this is really the case or am I missing sth.Yeah, that's what I meant when I wrote "Not sure I know anyone using it such that you protect against producer restart" since I never heard yet about a real Pulsar user using that feature.
Asaf Mesika
10/20/2025, 1:34 PMI understand Option 2 same as you do.
a
Artur Jablonski
10/20/2025, 1:44 PMI never heard yet about a real Pulsar user using that feature.Which makes me wonder. Is this because 1. People generally don't use Pulsar with strict effectively once requirements 2. People assume for effectively once they will need to have some sort of deduplication on Consumer side outside of Pulsar like in relational db or sth? 3. Sth else?
a
Asaf Mesika
10/20/2025, 2:16 PMI can only guess:
1. People dedup on the consume side, applicatively. Same they would have done in Kafka.
Asaf Mesika
10/20/2025, 2:17 PMI'm not even sure most Pulsar users read this Slack forum daily/weekly to view this message.
a
Artur Jablonski
10/20/2025, 2:22 PMI see... So is there a better place to shoot questions about application design around Pulsar?
4 Views