Hello I've got a large state Flink app that start...
# troubleshooting
v
Hello I've got a large state Flink app that started to restart because of this Kafka related exception:
Copy code
Caused by: org.apache.kafka.common.errors.ProducerFencedException: There is a newer producer with the same transactionalId which fences the current one
The exception is happening during the checkpoints (job configured as exactly once). What i cannot understant is how can that happen during a 2-3 minutes successful checkpoint if the
transaction.max.timeout.ms
is set to 1800000 (30 minutes) on kafka brokers and
transaction.timeout.ms = 480000
in Flink Kafka producers.
To be mentioned that the
transactional.id.prefix
is unique for 1 job only
Is there any other corner case situations where that could happen?
afaik on transaction should span across a single checkpoint only and there were no failed checkpoints before the exception poped-up
Would be very grateful for any hint 🙂
Also, mentioning that we use the old Flink kafka API
p
Did you manage to sort this one out?
v
Nope, it still happens sometimes