Hello :wave: I’m using Flink 1.14.5 and I’m trying...
# troubleshooting
d
Hello 👋 I’m using Flink 1.14.5 and I’m trying to understand what is affecting the trigger time of the checkpoints In my scenario: • Checkpoint interval: 1s • Minimum Pause Between Checkpoints: 500ms • Unaligned Checkpoints are enabled • Delivery semantic is exactly-once What is happening is that the checkpoints aren’t predictable, i.e, they can be triggered 5seconds after the latest ack checkpoint as can be triggered 10 seconds, 2 seconds and it is very rare to be 1 second Does someone have some insight of what I should pursue? 🙏
I already did some actions, such as increasing the memory/cpu of jobManager, reducing the sync/async duration in the tasks to accomplish the checkpoint, but I’m not seeing where I can see why the trigger time of the checkpoint is being affected or what can I do to expose the problem
g
You need to look for things like backpressure in your job, data skew, checkpoint size, checkpointing to s3. The Flink UI should provide some indication on these. Due to the above your checkpoint might take too long to complete.
d
Usually the job doesn’t have backpressure For instance, Trigger Time Latest Acknowledgement. End to End Duration. Check Data Size 2023-06-21 141958 2023-06-21 142000 2s 1.68 GB 2023-06-21 141937 2023-06-21 141939 2s 2.30 GB The job doesn’t have backpressure and the load of the operators is around 2% Looking into the history of checkpoints, the interval is 1s but the Flink Coordinator only sends the checkpoint in an unpredictable way
g
it might be due to checkpoint size being large which can result in longer checkpointing times.. especially if it uploads it to some kind of external dfs. You can also try enabling incremental checkpoints (i believe you are using rocksdb as a state backend)
d
Indeed, in this cluster I don’t have the incremental checkpoints enabled, but the end2end duration isn’t the time of the last subtask to complete the checkpoint? If the duration is 2s to complete, the new checkpoint takes 19seconds to be send … and I have other examples that take longer
g
do you notice any data skew between the tasks? can you also provide details about your environments and if you are using an external dfs for checkpointing?
d
there is data skew in some substasks, some of them process twice of the data
an you also provide details about your environments and if you are using an external dfs for checkpointing?
The checkpoints are being stored in S3 In this specific cluster, each TM • has one slot • JVM Heap memory is 13.9GB • Flink Managed memory is 13.9GB Sorry, the incremental checkpoint is enabled, in this job the configuration is enabled by the code
g
i suspect the culprit should be the checkpoint time for slower tasks and the time it take to upload it
d
I had the same suspicion but looking into the
End to End Duration
not sure if the problem might be in the JobManager. In this cluster, I only have one job running
g
The uploading to s3 is the TaskManagers job
d
yes right, my doubt is if the issue regarding the cadence of the checkpoint is in the JobManager (some missed configuration ?), because I’m not finding any evidence looking into the history of the checkpoint/logs that the issue is in the TaskManagers even in the presence of large state