In my current case for this job, it’s not configured to use the adaptive scheduler due to I believe another known issue (autoscaling causes state to be lost).
What’s happening in this case is the job checkpoints every 10s and the job itself is configured to use the autoscaling defaults (e.g. 5m window, etc.). The job gets the signal to scale and kills everything and results in a bunch of checkpoint failures. I’m not sure if / what the best configuration to change might be to help make this less noisy.