As per my Intro, we need to compile our models to several different platforms. We want to be able to set ray.remote options to target the right nodes (currently using Custom Resources , e.g.

resources={"A2":1}

) Currently it seems that this could be pretty easily achieved by (ab)using Hamilton's Tags feature:

@tag(**{"ray.resources": json.dumps({"A2": 1}))
def my_hamilton_node_fn_which_needs_an_A2(...) -> ...:
   ...

RayGraphAdapter.execute_node() would be modified to:

ray_options = {tag[4:]: json.loads(value) for tag, value in tags.items() if tag.startswith("ray.")}
return ray.remote(raify(node.callable), **ray_options).remote(**kwargs)

Any concerns with taking this approach? Any better options?

A overhaul we conducted a few months ago should make this usage pattern possible without hacking around too much 😁 Actually, we have

Lifecycle hooks

constructs that allow you to modify how Hamilton executes nodes via

.execute()

. These hooks have access to the node tags out of all things. So there should be a straightforward way to delegate to Ray! Here's some technical documentation to get you thinking. Tagging @Elijah Ben Izzy for additional thoughts

Hey — this is really cool. That’s a very reasonable approach actually, although we have a much more powerful way to do this if you’re feeling more adventurous. Let me first clarify my undersatnding: 1. You want to run certain nodes on ray (E.G. with resources) 2. The rest you want to run “locally” 3. You want to be able to specify the nodeds that are run on ray versus not 4. For those nodes you want to specify resources (parameters that you pass to

raify

, right? I like the approach you have actually (tags are reasonable for this), although you could easily add a separate

ray_remote

decorator that has a specific tag-format that the

RayGraphAdapter

will understand. Note, we now have more flexible execution as well — a whole system with a queue/scheduler optionally specifiable.

All nodes will run on Ray....we want some nodes to specify specific options (currently just 'resources' but makes sense to support any option supported by Ray) NodeExecutionMethod hook seems like a way of doing this without touching core code, however makes the code I write more complicated & it seems like making this the default Ray NodeExecutionMethod seems right to me if you're happy to use Tags for this....fully backwards-compatible (unless someone using ray. tags! Maybe ray_remote. would be a better namespace

Yeah, so tags are nice and clean — if you implement a

ray_remote

decorator it’ll just call out to tags. I think its pretty fair to add into the default, given it will be backwards compatible.

Thanks for posting Fran. I was going to follow up this week and ask about your Ray needs anyway 🙂

Yes #798 is what I meant...I think

ray_remote.

as the namespace matches the proposed

ray_remote

decorator, which I am very happy to work on implementing...I guess this should also live in the same file?

> I guess this should also live in the same file? Do you mean

h_ray.py

or

metadata.py

? If you put it in

h_ray.py

that will couple the hamilton code module to ray, i.e. you’ll need ray on the path to import the hamilton code. Is that acceptable? (gut says that’s okay). If you put it in

metadata.py

the hamilton code will still be “standalone”, but we now have a specific Ray decorator in the main libraray. Hmm. I don’t want to overengineer but specifying resources could be more general than just Ray. But we can always add a more generic one later.

I was thinking ray_remote decorator should live with the Ray code...but I get your point that then that means the user code couldn't be so easily used with a different driver so best in metadata

You could still use it with a different driver

That would be fine for me personally, however I think it violates Hamilton design

Yeah let’s go for metadata then.

This is as far as I got: https://github.com/DAGWorks-Inc/hamilton/pull/800 Needed to switch to using `.options(**ray_options)`due to the way Ray works, but that seems fine...not fully tested yet as it's a bit complex to set everything up my end & it's late here now (am in UK) but I wanted to post the WIP I had so far in case it throws anything up.

cool thanks! Will look over by the time you get in the morning hopefully 🙂 or maybe during office hours tomorrow.

Chiming in, thanks for the PR, this looks great! Cool thing is that this can be used in both APIs (the task-based and the node-based ones). I’m in favor of putting this in the

h_ray

code, it keeps it decoupled. Then we can add a more general one? I’m also happy to keep the

ray_remote

namespace. If the problem is

reserved_forbidden

being passed through we can bypass that at the tag constructor level, and make

_key_allowed

not static (there’s no reason it should be). This should be easy, anything ending with an

_

is a reserved

kwarg

, so you can change the

tag

kwarg to add

bypass_reserved_namespaces_=False

, then have

ray_remote

delegate to the

True

value of the constructor argument.

I think the only thing outstanding on this now is Docs: https://github.com/DAGWorks-Inc/hamilton/pull/800 Let me know which docs you think we need, beyond the code comments 😉 Also if there's anything else that you think needs doing to get this merged.