Hello :wave: Has anyone had issues with configuri...
# troubleshooting
j
Hello 馃憢 Has anyone had issues with configuring the autoscaler with jobs that have large state (over 10mins, sometime up to 20mins recovery/checkpoint time) and having the autoscaler kick in during recovery/checkpoint process - I鈥檓 wondering if anyone has found a way to prevent such behaviour ? Also; these jobs takes a long time to scale because of state data redistribution (basically doing re-shuffle of the whole state data) every time. The best we found for that is to set the default parallelism close to max but still becomes an issue when throughput picks up and we have to go through 2-3+ round of scaling out.
m
You can set max-parallelism higher as well as
job.autoscaler.scale-up.max-factor
to 6-10 to scale up faster when workload increases. To react to workload increase faster you may need to adjust your
job.autoscaler.metrics.window
to shorter interval. I'm actually facing an opposite problem - it doesn't scale down when workload is 0 after all pending records had been processed. I'm not sure if the problem is due to
job.autoscaler.scaling.effectiveness.detection.enabled
=
true
or
job.autoscaler.restart.time-tracking.enabled=true
-- maybe it tried once and then saw that it's taking too long so not worth scaling down, or may be there was an error on scaledown and it rolled back to the last known good parallelism due to
kubernetes.operator.deployment.rollback.enabled=true
Large state is a problem, yes. I was getting timeouts on scaledown. The solution had been to increase JobManager CPU and reduce scaledown factor.
j
For the job manager ? interesting never thought that would be a factor. We definitely had some issues with task manager network bandwidth saturation but tuned some rocksdb network thread + topologySpread and now are doing 250mb/s on average; that helped quite a bit.
m
My understanding is that JM CPU is required to communicate with all the TMs it's going to create on scaling op; not enough and you get TM timeouts.
j
I鈥檒l definitely monitor that on my next tries
馃憤 1
m
You may also try increasing
hearbeat.timeout
- I see a lot of job restarts due to that value being too low and causing an error, especially when there are dozens or 100+ task nodes, at least in my environment.