Grzegorz Liter
03/11/2025, 2:05 PMorg.apache.kafka.common.errors.InvalidPidMappingException finally we were able to crack the issue:
https://issues.apache.org/jira/browse/FLINK-37460
The lesson from this is that when using State Processor API with Exactly Once KafkaSink one have to clean KafkaCommitable state.Martijn Visser
03/11/2025, 2:07 PMMartijn Visser
03/11/2025, 2:08 PMGrzegorz Liter
03/11/2025, 2:08 PMGrzegorz Liter
03/11/2025, 2:09 PMfor (CheckpointCommittableManager<CommT> manager :
committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) {
It won't load the lingering KafkaCommitable until we hit again checkpoint 10000 but when we hit it the stored info about tx is already expired in Kafka Cluster and causes exception.Martijn Visser
03/11/2025, 2:11 PMGrzegorz Liter
03/11/2025, 2:11 PMMartijn Visser
03/11/2025, 2:12 PMGrzegorz Liter
03/11/2025, 2:12 PMMartijn Visser
03/11/2025, 2:12 PMGrzegorz Liter
03/11/2025, 2:12 PMGrzegorz Liter
03/11/2025, 2:14 PMGrzegorz Liter
03/11/2025, 2:29 PMcommittableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) reads only up to lastComletedCheckpointId? If that would load all commitables that are in state this would also not happen.
How can it have normally commitable for future checkpoints?
I guess when using Unaligned Checkpoints? I never used those.