Hi all, I have a question about incremental snapshotting. I have been using incremental snapshotting with RocksDB as the state backend. However, I’ve noticed that even though I’ve configured incremental snapshotting, the size of the last checkpoint periodically matches the size of a full snapshot. Just like the image below. I thought incremental snapshots would only include changes from the last snapshot, so I expected their sizes to be smaller. Is this expected behavior? Has anyone experienced something like this or can help me understand why this is happening? Any thoughts or suggestions are welcome. Thanks in advance. Lastly, I will add my flink configurations in the below. flink version: 1.16.1 “state.backend”: “rocksdb”, “state.backend.incremental”: “true”, “state.checkpoint-storage”: “filesystem”, “state.checkpoints.num-retained”: “10" “execution.checkpointing.mode”: “EXACTLY_ONCE”, “execution.checkpointing.interval”: “30s”, “s3.endpoint”: “{s3_endpoint}“, “state.checkpoints.dir”: “s3://flink-checkpoint/checkpoints/****”, “state.savepoints.dir”: “s3://flink-checkpoint/savepoints/****,

bro, although I can't help you solve the problem, I am more interested in your architecture design. I want to refer to what your architecture is like, and share it if it is convenient. ths😗

Hi there, I’m not really sure if this is what you’re asking for, but I’ll provide a brief overview. It’s a typical kafka to kafka streaming application deployed to k8s, which has s3 as an checkpoint storage and prometheus for collecting metrics. We are currently on staging test right now to see if we can operate this app as we expected. 😀

Hi @Dongwoo Kim did you ever figure out why this happened. Im seing the same issue.

Hi @Ole Bille , this was due to RocksDB’s periodic compaction. RocksDB periodically triggers compaction and creates one bing .sst file. And when flink uploads this big file it is uploading a full snapshot. To mitigate this issue we are using changelog statebackend. This approach allows us to reduce the frequency of uploading full snapshots while creating more small files as a tradeoff. ref) https://flink.apache.org/2022/05/30/improving-speed-and-stability-of-checkpointing-with-generic-log-based-incremental-checkpoints/

Thx Dongwoo Kim for replying. Ill take a look at it. Sounds like its the same issue I have.