Hello, Does anyone know why a checkpoint of 512MB...
# random
j
Hello, Does anyone know why a checkpoint of 512MB size takes approximately 10 seconds to complete using a hashmap, but nearly 150 seconds with the RocksDB state backend? This is observed in the same application, where the RocksDB version has twice the CPU and memory allocation.
c
My first guess is that you have a slow disk. From the docs: "_With RocksDBStateBackend, in-flight state is first written into off-heap/native memory, and then flushed to local disks when a configured threshold is reached._" https://flink.apache.org/2021/01/18/using-rocksdb-state-backend-in-apache-flink-when-and-how/
gratitude thank you 1
j
Thanks, that makes sense. I'll see what I can do in that regard.
s
You can also check which operator is taking the most amount of time in flink UI or the metrics for the job. Usually, if the task is under pressure and is not able to process efficiently, then the checkpoint takes up time to finish.
πŸ‘ 1
gratitude thank you 1
a
We observed that jobs with a lot of in-flight data were very slow in making snapshots. Try limiting in-flight data. Add parallelism ot make buffers smaller.
gratitude thank you 1
j
Thanks, that was exactly the case, after reducing the size of in-flight data checkpointing got faster
πŸ‘ 1