Hello,
Does anyone know why a checkpoint of 512MB size takes approximately 10 seconds to complete using a hashmap, but nearly 150 seconds with the RocksDB state backend? This is observed in the same application, where the RocksDB version has twice the CPU and memory allocation.
Thanks, that makes sense. I'll see what I can do in that regard.
s
Shubham Bansal
03/26/2024, 4:21 PM
You can also check which operator is taking the most amount of time in flink UI or the metrics for the job. Usually, if the task is under pressure and is not able to process efficiently, then the checkpoint takes up time to finish.
π 1
gratitude thank you 1
a
Ari Huttunen
03/29/2024, 1:04 PM
We observed that jobs with a lot of in-flight data were very slow in making snapshots. Try limiting in-flight data. Add parallelism ot make buffers smaller.
gratitude thank you 1
j
Jasmin Redzepovic
04/02/2024, 7:56 AM
Thanks, that was exactly the case, after reducing the size of in-flight data checkpointing got faster