Hi guys,
the hamilton ray adapter is using Ray Wo...
# generalv
Volker Lorrmann
03/26/2025, 10:06 AMHi guys,
the hamilton ray adapter is using Ray Workflows. The ray docs mentions, that Ray Workflows is deprecated and will be removed. https://docs.ray.io/en/latest/workflows/index.html
Are there any plans to update the hamilton ray adapter?
Thanks!
s
Stefan Krawczyk
03/26/2025, 6:11 PMSo there's two adapters.
Stefan Krawczyk
03/26/2025, 6:12 PMRegular Ray and an old ray workflow based one.
Stefan Krawczyk
03/26/2025, 6:13 PMSo we just need to delete the workflow one -- if you wanted to submit a PR? 😉
v
Volker Lorrmann
03/26/2025, 6:15 PMThanks.
What is the difference between these?
The ray workflow based one converted every node into a ray remote function and the new one creates one remote function/module for the whole dataflow?
s
Stefan Krawczyk
03/26/2025, 7:14 PMBoth do the same in terms of turning functions into ray.remote ones. Workflows just auto did checkpointing on top of that IIRC.
v
Volker Lorrmann
03/27/2025, 8:44 AMOk. Thanks
Volker Lorrmann
04/02/2025, 7:23 PMJust found, that beside the adapters, there is also the
RayTaskExecutors
Stefan Krawczyk
04/02/2025, 7:32 PMThe task executor is used when you are using the parallel and collect constructs. It means that a whole Hamilton subgraph becomes run as a single Ray task.
Stefan Krawczyk
04/02/2025, 7:34 PMThis is useful if you're doing the same thing over and over and want to parallelize that over Ray. E.g. file processing
Stefan Krawczyk
04/02/2025, 7:35 PMThe ray graph adapter instead makes each function a ray remote task. Right now you can only do one or the other. No inherent limitations, just that's what was implemented, e.g. we could make the adapter smarter and work with the parallel and collect constructs...
v
Volker Lorrmann
04/02/2025, 8:06 PMThanks Stefan! So in general, remote executors are used for parallel and collect and adapters like RayGraphAdapter, or FutureAdapter are used to parallelize all nodes?
s
Stefan Krawczyk
04/02/2025, 8:10 PMyep that's a reasonable summary.
Stefan Krawczyk
04/02/2025, 8:10 PMWe could get smarter about grouping parts of the DAG into tasks for Ray to get around SERDE costs - but that's a bit of a research topic.
🙏 1