Hello. Does anyone know what is the proper way to set up ForST to actually do a fast savepoint. Currently its at least 30 times slower than rocksdb. Im testing enabling s5cmd fast-duplicate. But still nothing

which storage are you using for savepoint?

S3

Hi Francisco, Youc can try these points as well: 1. Verify fast-duplicate is actually triggering — enable DEBUG logging on org.apache.flink.state.forst and org.apache.flink.runtime.state. If you see byte-level copies instead of path-level copies, that's your 30x problem. 2. Don't set a separate state.backend.forst.primary-dir — keep it as the default (__checkpoint_dir__) so the path validation for fast-duplicate succeeds. 3. Use retained checkpoints as your "savepoint" mechanism — ForST incremental checkpoints reuse SST files and are dramatically faster. Savepoints with NO_SHARING will always be slower because they must copy everything. 4. s5cmd helps but isn't the silver bullet for savepoints — s5cmd accelerates file transfers, but the real issue is that savepoints don't reuse files. The fast-duplicate (S3 CopyObject) path is what matters most. 5. Consider upgrading to Flink 2.3 when available — FLIP-555's native S3 with CRT client and bulk copy support should significantly improve ForST savepoint performance. 6. The hardcoded 4-thread limit in ForStStateDataTransfer is a known limitation (marked with a TODO in the code). There's no config option yet to increase parallelism.

I noticed fast duplicate was triggering for checkpoints but for savepoints as it was native format it seems to just recreate the entire savepoint but i will test this configurations. My only concern with point 2 is that this would lead to potential duplicates whenever we need to update the application, if we are recovering form last checkpoint

No issues. Please share your findings as it will be helpful for me as well, as I was also doing some poc on forst but had to put it on hold for some time due to some other task. Thanks @Francisco Morillo.