Hi All! What is the recommendation to avoid infinite storage growth in my S3 bucket that holds my checkpoints? I want to have a lifecycle policy to clean up old SST files (RocksDB with incremental checkpoints) but I find those lifecycle rules still sometimes delete files that are part of my checkpoint. Meaning some SST files never get re-written for very long periods of time but are still part of new checkpoints. We are trying to set

state.backend.rocksdb.compaction.filter.periodic-compaction-time

, but after some reading it seems this setting still doesn't quite do what we want.

I created a script that looks at the checkpoint metadata files, looks for the files it references, and deletes all of the sst files that are not referenced there, and was created before the script was run (to avoid deleting new files from newer checkpoints).

This seems like something that Flink should just be better at. It should be pretty easy to have a feature that is simple: • when a checkpoint is taken: ◦ If an SST file age > x ▪︎ re-write. Maybe that's what

state.backend.rocksdb.compaction.filter.periodic-compaction-time

does, and it seems like it should, but isn't 100% clear that's it.

Hello Darin, which connector are you using for this?

This is for checkpoint persistence in S3, using the Presto plugin. (s3p)

I don't think the compaction can help you with this one. What we noticed is that for some deployments, specifically ones that explicitly start from a previous explicit checkpoint, it starts a new job and "detaches" from all previous sst files. That means you will have orphan sst files inside your checkpoint dir, that won't be compacted because it is no longer owned by the job.

For those files the S3 lifecycle rule eventually picks them up and removes them. That's not the concern. The concern is making sure the lifecycle rules never remove files that are still in the checkpoint.