This message was deleted.

Single Threaded Producer is likely the issue. Try having multiple threads produce to the same partition.

I need to ensure the order of data within a single partition, so I can only use a single producer.

Can you send the messages async? - https://github.com/apache/pulsar-client-python/blob/766db9e0420954798d8d24ef4dc55fa250270921/pulsar/__init__.py#L1100 The send retuns when it is Acknowledge. So it is slower

I have tried using

send_async

to send data, and the QPS can reach 30,000, but I’m worried that the data will be out of order. I don’t understand why

send

is so slow.

Send waits for the ACK from the broker. It will be ordered if there is only one thread.

I can achieve fast performance when using

ack='all'

in Kafka as well.

The problem is that as your python is blocked until one message is ACK by the broker. So the slowness is the network connectivity between your cient and the broker. In your use case, sending async with just one thread will avoid the blocking of the thread and it will be keep the order. there is no ack='all' on the send in Pulsar. You have to use

send_async

If three messages are sent using

send_async

, and middle of them fails to be processed by Pulsar, will the data be lost or unordered? I

Is it normal for the QPS of the send() method to be around 500

yes. 🙂. each send is incurring the network round trip time cost. it has nothing to do with pulsar. kafka's send_sync() will perform the same. How are you achieving ordering in kafka with high throughput? As far as I remember, Kafka client can also reorder messages due to its retry policy in the face of failures. Check https://kafka.apache.org/documentation/#producerconfigs_retries. This should limit throughput.