Is there a known configuration or way to handle ch...
# troubleshooting
r
Is there a known configuration or way to handle checkpoint failures during autoscaling events? I have a specific job that is configured to aggressively checkpoint (every 10s) however during an autoscaling event, I’ll see those checkpoints fail on account of the sudden autoscaling. It’d be nice if it could wait for the checkpoints to finish instead of failing when the job autoscales or perhaps it’s just a misconfiguration. Any recommendations here?
g
There is an ongoing Flip discussion about improving rescaling to happen right after checkpoints to make this nicer
To synchronize it with checkpointing in other words
r
Thanks @Gyula Fóra ! Do you have a link to that by chance?
Assuming it’s this maybe?
In my current case for this job, it’s not configured to use the adaptive scheduler due to I believe another known issue (autoscaling causes state to be lost). What’s happening in this case is the job checkpoints every 10s and the job itself is configured to use the autoscaling defaults (e.g. 5m window, etc.). The job gets the signal to scale and kills everything and results in a bunch of checkpoint failures. I’m not sure if / what the best configuration to change might be to help make this less noisy.