pulsar

Hi! Is there anything blocking a 3.9 or 3.10 release for python client on both mac and linux besides the version constraint in the build?

<https://github.com/tspannhw/FLiPN-Py-Stocks|https://github.com/tspannhw/FLiPN-Py-Stocks>

I don’t recall any such connections errors in 2.10.x. Though, did you try to upgrade to 3.1.0, just in case?

i downgraded python for now, but will give 3.1 a shot sometime

Reconfirmed, pulsar producers from 2.10.x do not work in python 3.9.x
3.1.x fixes the issue

batch receive is not yet supported in Python, though keep in mind that client library is already batching internally both the delivery and the acks.

when you call ack, the client will by default group the acks

How is ack grouping performed? I.e. if my client calls `acknowledge`, did that packet leave the host and make it to Pulsar?

I know acks are conceptually "unicast", but I'd hope that the act of `acknowledge()` returning at least indicates that a TCP ACK was received from upstream and that I/O for the unicast-send was complete.

<@U0215TMGBDH> there’s no really a way to know when a tcp-ack is received (using Socket abstraction).

The grouping is configured in consumer config. There’s a time delay: eg: 100ms. All the acks performed in that window are sent in one single shot to the broker, to reduce the RPC interactions.

That's really unfortunate, and the premise is also semi-incorrect:
• If all you're doing is a `write(2)`, a blocking write on a berkeley TCP socket indicates a _high affirmative probability_ of an ack being received.
• But, more importantly, doing a write gives a guaranteed _negative_ confirmation; if the upstream's TCP stack is not online, my write will not succeed. 
• Unbuffering the socket can mitigate the probability dimension.


At the application level: ack-acks (broker-&gt;client) are highly desirable in this area, and are already supported by the Java client.

Even absent app-level acks, using _intentional_ buffering of the writes with such a large window (I can process a lot of messages in 100ms) seems like a recipe for trouble.

Like, I get the at-least-once/at-most-once tradeoff. But being able to present my application's users the firm choice between "your data *definitely will not be redelivered* if you make it past the line that says `acknowledge` without an exception being raised" and "your data *may or may not* be redelivered if you don't make it to that line, or if that line raises or blocks and your program crashes" is really valuable to them.

The ability to do that is worth, not exaggerating, a 50% loss in throughput or 150% broker capacity need for me.

A good way to do this might be "barrier" invariants, i.e. "the next time a batch of messages is sent from the broker to the client, the broker's doing so implies that acks for all previously acknowledged messages have been successfully processed". Of course, to do that, I'd need clients to indicate whether a reconnect happened internally between `receive` calls, and what monotonic time acks were sent with vs. messages were pulled into the local buffer with.

AMQP uses the barrier model here. I *think* the Kafka transaction system uses a sequence identifier internally, but am not as familiar with that client's code.

&gt; If all you’re doing is a `write(2)`, a blocking write on a berkeley TCP socket indicates a _high affirmative probability_ of an ack being received.

Not really though. It would just block until it has space in the TCP receiver window. That doesn’t even mean the packet has made it out of the same machine.

&gt; • But, more importantly, doing a write gives a guaranteed _negative_ confirmation; if the upstream’s TCP stack is not online, my write will not succeed.
That is true. We used to do this in the earlier version of Pulsar client. Ack was a blocking operation with sync and async version, which would complete as soon as the ack was written in the TCP connection.

The problem was that it was *very* confusing for users.

People would not expect an error in the ack (even though the signature could throw exception.. ) whenever a broker was restarted.  More importantly, they in general wouldn’t know exactly how to react to an ack failure.

eg: should I retry the ack? should the client library already have retried? or what else can the application do?

The only reason for an ack to fail is that the connection with the broker was broken (eg: broker restarted) or that the topic moved to a different broker.

Per the semantic, the message is already going to be delivered again, which will cause a 2nd process and the ack should go through.

In most cases, that’s the best and easier way to deal with ack failures --&gt; do nothing.

<@U0215TMGBDH> BTW: this was merged recently <https://github.com/apache/pulsar-client-cpp/pull/232> and will allow to wait for the ack-on-the-ack

Ah, fantastic! Once the python client gets it, that'll be super useful.

Hey guys, I’d like to know if the transactions API can become available in Python. Are there plans on it this year? If not, how feasible is it to be implemented in Python?

Can you get the tags that pip is looking for? eg; `pip3 debug --verbose`

You can also compare the tags with the tagged release files here: <https://pypi.org/project/pulsar-client/3.3.0/#files>

For debian arm64 on python 3.7, it should be using this file: <https://files.pythonhosted.org/packages/5d/45/4c326a2396afe702bc254da06e0336048da5f86c1d3453a83f10411f4364/pulsar_client-3.3.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl|pulsar_client-3.3.0-cp37-cp37m-manylinux_2_17_aarch64.manylinux2014_aarch64.whl>

Thanks Matteo - it's weird when I added a step to upgrade pip (after installing it) - it now works - odd but hey that's Python :slightly_smiling_face:
``` =&gt; CACHED [stage-0 43/48] RUN apt-get install -y python3-pip                                                                                                                            
 =&gt; CACHED [stage-0 44/48] RUN pip3 install numpy                                                                                                                                        
 =&gt; CACHED [stage-0 45/48] RUN pip3 install --upgrade pip                                                                                                                                
 =&gt; CACHED [stage-0 46/48] RUN pip3 install pulsar-client==3.3.0```