Hello Everyone, Today i have a very unique issue. We are dealing with issue with Flink Restart. We are using Kinesis, Rocksdb and dynmodb as our infrastructure. Here everything is working fine, we can see that records are fetch, processed and pushed to dynamodb, until there is a restart. Following thing we have observed when restart occurs. 1- There is constant back pressure 2- Nothing In/out buffers for like 20min before restart 3- no resource usage, CPU usage is close to nothing 4- I see checkpoints faling at 20 minutes before that 5- Batch write to dynamodb is constant in that window. Sharing your monitoring result as well. We are still not sure what can be the cause of issue, any suggestions would really help.

What does the Flink UI suggest for the checkpoint duration and size?

This is the size of checkpoint, we are using unalligned checkpoint.

what about watermarks? do you see watermarks progressing?

Not sure about the time the issue happens because its kinda random.

can you provide some more context on what the pipeline actually does and if you are using processing or event time?

So we are basic fetching events, processing using hash map and then save in dynamodb. What we have seen that 20 min before restart i.e. when checkpoint start to fails we stop consuming as you can see in screenshot as well. And soon as we restart we get back to track and reconsume it again.