Anyone familiar with exactly-once processing/transactions around? I have a Kafka/Flink question that someone might be able to help with. I noticed that I was encountering a number of failed/missing transaction ids in one of my pipelines. When I started digging a bit deeper, I realized that the internal

__transaction_state

Kafka topic only had a default retention of an hour (

<http://retention.ms|retention.ms> = 3600000

), however I have several timeouts defined in my Flink jobs related to transactions that are greater than this default timeout (

<http://transaction.timeout.ms|transaction.timeout.ms> = 86400000

). Since transactions are ultimately backed by Kafka and I’m assuming the

__transaction_state

is it safe to assumed I’d need the retention for that topic to match my maximum timeout? I suspect this could explain some of the frequent expired transactions I’ve been encountering, but wanted to try and verify it.

The Immerok content we had for this is temporary offline, but the Wayback Machine might help for now https://web.archive.org/web/20230201061412/https://docs.immerok.cloud/docs/how-to-guides/development/exactly-once-with-apache-kafka-and-apache-flink/

I’m guessing that the

<http://transaction.max.timeout.ms|transaction.max.timeout.ms>

corresponds to what would be the retention for the

__transaction_state

topic? We’re using the Strimzi operator for Kafka, so I just want to make sure that I’m setting the appropriate values in the appropriate places.

That would be a guess for me too 😓

Hehe, well this is super helpful though. I know we’ve chatted multiple times about those stupid

InvalidMappingPidExceptions

or whatever they are, so seeing this makes a lot more sense.