Hi Team, I am setting RocksDBBackend path to a persistent storage (EFS). When new state is stored in storage, the folders created are in the format:

job_<jobid>_op_KeyedProcessOperator_<operatorid>__<taskIndex>_<totalParallelism>__uuid_<randomuuid>/

Here the folder name changes every time the job restarts. Also the folders seem to be deleted after restarts. How do I make sure the state is retained and when I restart the job, it is able to query the state. Appreciate any inputs on this. Thanks

By restart you mean you cancel the job and start it again right? to keep it around when u cancel the job, use this. The folder name changes because a new job starts every time. You can stop the job by taking a savepoint and restart your job from that savepoint

Also, using a EFS for your storage backend is an anti-pattern, you’ll experience severe performance impact.

yes @Giannis Polyzos, We have streaming jobs. If we delete the deployment and deploy again, or if we delete the pods and it spins up again, we want it to continue and also query the state it had earlier. Is checkpoint retained when the job is redeployed ? or only save-points are retained ? hey @sap1ens, oh, dint know that. What would you suggest instead of EFS for persistent storage ? Just to add, we are using the EFS path for RocksDB storage path and s3 for check-pointing path

oh, dint know that. What would you suggest instead of EFS for persistent storage ?

Just rely on S3 for failure recovery (using checkpoints / savepoints). And for the actual RocksDB backend you need to use the fastest SSD drive possible. Even EBS disks can affect performance, so it’s better to use instance-level storage. Yes, that’s not persistent, but that’s why you’re using S3 for failure recovery.

Got it. Thanks @sap1ens. Any inputs on this scenario. Are we missing something here ?

How would it be able to read the state for the previous checkpoints ?

Short answer: you don’t 🙂 It’s possible, but it can become a big headache, because checkpoints are usually not self-contained (I assume you’re using incremental checkpoints). It’s better to use savepoints for redeployments instead.