We had recurring problem with `org.apache.kafka.co...
# troubleshooting
g
We had recurring problem with
org.apache.kafka.common.errors.InvalidPidMappingException
finally we were able to crack the issue: https://issues.apache.org/jira/browse/FLINK-37460 The lesson from this is that when using State Processor API with Exactly Once KafkaSink one have to clean KafkaCommitable state.
m
I'm not sure that I would classify this as a bug tbh: the moment you're going to use the State Processor API to manually change the internals during the 2-phase commit process of Flink towards Kafka, you're fully in internals land that's not a user facing construct
g
The problem happens without messing with internals of 2-phase commit. Problem is that State Processor API drops checkpoint Id to 0. So the KafkaCommitable keeps transaction for checkpoint e.g. 10000 but when we start job from migrated state we start from checkpoint 0
Then here: org.apache.flink.streaming.runtime.operators.sink.CommitterOperator#commitAndEmitCheckpoints
Copy code
for (CheckpointCommittableManager<CommT> manager :
                    committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId)) {
It won't load the lingering KafkaCommitable until we hit again checkpoint 10000 but when we hit it the stored info about tx is already expired in Kafka Cluster and causes exception.
m
Yeah, because imho when you're doing exactly once processing, the checkpoint Id shouldn't be reset to 0. Else, everything will have to be checked in order to validate if each event has been processed already, or needs to be aborted/completed.
g
100%. So one of the solution for raised issue would be to make State Processor API to carry over checkpointId which it doesn't
m
Yeah so then I would say the bug would belong in State Processor API land, not in the Runtime land. The runtime works as intended
g
True, I agree
m
Cool cool, I'll change the affected component then πŸ™‚
g
just did that
For mitigation we can ether: 1. change UID of Sink so the state of the sink is not loaded 2. remove (by changing UID becasue there is no delete API in State Processor API) the Commitable state So the 2PC is not trying to commit transacion that is already commited anyway (giving successful job termination with savepoint)
Btw. why
committableCollector.getCheckpointCommittablesUpTo(lastCompletedCheckpointId))
reads only up to lastComletedCheckpointId? If that would load all commitables that are in state this would also not happen. How can it have normally commitable for future checkpoints? I guess when using Unaligned Checkpoints? I never used those.