Hello,
We're having some issues where continuously running flink jobs restart and begin processing from hours ago. The events are pulled off a Kafka topic. We write checkpoints to s3 every 5 minutes - but when we enter into these failures, and all task managers exit and restart, the recovering the jobs load the HA and pick a checkpoint which is often hours old. Not the latest.
What are we doing wrong?
We'd like to reduce reprocessing any events, and really keep backpressure under 30 minutes ideally, definitely under an hour.