Hello team, I've successfully implemented the RichAsyncFunction, which is supposed to provide full exactly-once fault tolerance guarantees, as stated in the documentation. However, I'm encountering an issue where events are being duplicated despite this guarantee. You can find the documentation link here: https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/datastream/operators/asyncio/#fault-tolerance-guarantees Here's my scenario: I have a Flink job that processes two events. After successfully processing the first event, the job encounters a failure. Upon restarting the job, it should only process the second event. However, both events are being processed again, leading to data duplication. The checkpoint interval is set to 1000ms. Thank you for your attention to this matter. Looking forward to hearing your thoughts and recommendations.

Have you enabled exactly once?

While using

KafkaSink

it is possible to set the _`setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)`_ but I want the sink function to be asynchronous. So to overcome that I have used

RichAsyncFunction

because in the documentation it is stated that it provides exactly once transaction. Just FYI, I am using Redpanda.

I don’t think it states it provides exactly once transactions, but guarantees: the purpose of the function is for use cases such as enrichment.

Okay, got your point. So I have observed that if I use

KafkaSink

, it degrades the performance. So Is there any way to use _`EXACTLY_ONCE`_ with better performance?

Which parts of “performance” are you looking to improve?

Regarding the sink function, I previously utilized KafkaProducer to handle data sinking, which offered better performance compared to using KafkaSink.

Given that the FlinkKafkaProducer had some bugs which are resolved on KafkaSink, it could also be that you didn’t have exactly once back then

I am not using FlinkKafkaProducer but KafkaProducer. Performance-wise, when utilizing KafkaProducer, my job was able to easily handle 10k EPS with 23 sets of rules on a 4-core CPU and 4 parallelism. However, upon implementing KafkaSink with the EXACTLY_ONCE property configured, the sink function of my job became 100% busy under the same conditions of rules, eps, and CPU.

100% busy means only that’s it not idle or backpressured

Okay, thank you @Martijn Visser for your help. I will look into it. 👍