In my application logs reading from and writing to kafka with exactly once enabled, after my upgrade to Flink 1.14, I find a lot of those

INFO

log messages:

ProducerId set to -1 with epoch -1

and

ProducerId set to 2359014 with epoch 0

. Investigating a bit into this shows me that those log events are no errors, can be ignored and indeed are on purpose polluting my logs 🙂 In an old flink mailing list post Tzu Li (Gordon) wrote:

With that in mind, when using exactly-once semantics for the FlinkKafkaProducer, there is a fixed-sized pool of short-living Kafka producers that are created for each concurrent checkpoint. When a checkpoint begins, the FlinkKafkaProducer creates a new producer for that checkpoint. Once said checkpoint completes, the producer for that checkpoint is attempted to be closed and recycled. So, it is normal to see logs of Kafka producers being closed if you're using an exactly-once transactional FlinkKafkaProducer.

which more or less describes what I see in Flink 1.14 codebase despite being an older post: For each new checkpoint, Flink will reuse a producer, but assigns a new transaction.id leading to reinitializing a new transaction-id (producer-broker roundtrip) and assigning a new producerId and epoch. My question is: Why does flink do this? From a Confluent blogpost about transactions, the "default" way to use transactions seems to assign a transactional.id once with application start and keep it from then on. I understand that the pattern used by flink works as well, but is there a specific purpose on why creating fully new transactional ids with each checkpoint having a fully runnig application? Was it easier to implement? Or are there details requiring flink to work that way? Is there a design doc or something where thoughts around all the internal stuff with regards to the Flink Kafka source/writer are shared?

@Theo Diefenthal This message is better suited for #C03G7LJTS2G and not for this channel

But I'm not having trouble, just asking for design information 🙂 Thougt #C03G7LJTS2G is more for people with urgent issues blocking them..

I can understand, but the description for random is

Asking for help on using Apache Flink!

- I think your message is related to Flink so fits there 🙂

Alright then, thanks 🙂