I’m having an issue with Flink’s checkpoints—speci...
# troubleshooting
m
I’m having an issue with Flink’s checkpoints—specifically, the checkpointing time has suddenly increased since yesterday. Previously, it took around 3-4 seconds, but now it takes at least 30 seconds, and sometimes even up to 8 minutes to complete a checkpoint. This is happening even though I haven’t made any changes to my job, and the disk status appears normal. My checkpoint interval is set to 10 seconds, and the pause time between checkpoints is 1 second. Can anyone guide me on how to diagnose this problem?
Image from iOS.jpg
g
checkpoint interval and min pause are too low - if you are running a stateful job.. What does the job do? Do you have unaligned checkpoints enabled? As more are more state keeps accummulating, checkpoints should start becoming slower and while the job waits for them to complete, it will start adding backpressure.
m
@Giannis Polyzos My job is stateful. I set the checkpoint interval to 10 seconds because my checkpoints usually took around 4 seconds, but suddenly the time has increased a lot. My job has two processors: 1. The first processor joins two topics and sends the result to the second processor. 2. The second processor joins the result with another topic and produces the final output. In the first processor, I have two ValueStates, each holding about 35 million records. In the second processor, there’s a MapState that also stores around 35 million records. I didn’t enable unaligned checkpoints because it’s very important for me not to lose any data.
g
So it's a 3 way join here.. 1-3 minutes is the suggested checkpoint interval. Everytime you take a checkpoint you job "pauses", and the larger the state grows the backpressure starts building up... taking a read here might help give some more context
❤️ 1
m
Thanks a lot
One more question, Is taking a checkpoint in stateful jobs more costly than in stateless jobs? If so, why?
g
because of states.. the larger the state the longer it takes to upload the data to remote storage.. stateless jobs have no state
👍 1