Hi team, we just upgraded kuberay and ray from

2.35.0

to

2.53.0

. We are using autoscaling clusters. One thing I am noticing is that the ray cluster is not scaling to the same number of workers after changing, leading to longer runtime. There is more work that could be done by scaling up, but seems like daft doesn't know this? Any way I could get it scale up more workers through some configuration or partitioning upfront? Trying to avoid manual scale up/down.

Hey Garrett, could you try setting

RAY_enable_autoscaler_v2=0

to see if that helps? It would help us triage what's going on

let me give those a try and see what happens

Hmm will need to take a closer look at our autoscaler to confirm if this is an issue on our end. As a temporary unblock (and to help us confirm the diagnosis), could you try calling

request_resources

directly from your driver code before running the pipeline? This bypasses Daft's autoscaling logic entirely and tells Ray's autoscaler the full demand upfront:

import ray
from ray.autoscaler.sdk import request_resources

# init ray...

# Manually scale
request_resources(num_cpus=TOTAL_CPUS_AT_MAX_SCALE)  # e.g. 10 workers * CPUs per worker

# then run your Daft pipeline as normal

If the cluster scales to 10 workers and runtime matches what you saw on 2.35.0, that confirms the issue is on our end. Would be useful to know either way.

will give that a try tomorrow