Hi Folks, I am repurposing a fairly large DAG that used to run locally, to deploy it remotely on Ray. This is super easy with the adapters provided. I have run into two performance bottlenecks though. First, I move polars dataframes that can have a billion rows. I have noticed that when attaching a

HamiltonTracker

up to 20% of my DAG tuns can be spent calculating dataframe stats for display in the UI. I have short-circuited the calculations by overriding

hamilton_sdk.tracking.polars_stats._compute_stats_

by an empty function, but any less hacky suggestions are welcome. Second, and more crucially, I hit a very steep serialization penalty when moving data between nodes. If I understood correctly, we can eliminate serialization (at the cost of parallelism, possibly) by putting all the nodes in the same group, but I can’t seem to pull this off. Here’s my code:

class MyGS(grouping.GroupingStrategy):
    def group_nodes(self, nodes: List[Node] -> List[NodeGroup]:
        group = NodeGroup(base_id=“whatever”, spawning_task_base_id=None, nodes=nodes, purpose=NodeGroupPurpose.EXECUTE_BLOCK)
        return [group]

Then I just attach this to my driver by calling

driver.Builder().with_grouping_strategy(MyGS())

However, this doesn’t seem to have the desired effect, stuff still gets serialized. Any suggestion what I am doing wrong? Or another possible workaround?

hey — can you share how you’re running it? The grouping strategy doesn’t work with the standard ray adapter. TBH you may be better served running the whole thing on Ray, but depends on grouping.

I literally just add a

RayGraphAdapter

and then execute the DAG as usual. Not sure the code will be informative, but can provide it if useful. I don’t insist on grouping, just want to avoid serialization. If possible.

Got it — yeah will provide instructions but the grouping strategy doesn’t currently apply for the graph adapter (sorry it’s a bit confusing…)

I want all in one. Could just create a driver remotely, but figured there should be a better way. But serialization time to disk ~ runtime so I might still end up with that. EDIT: Not sure that I answered your question. Really my point is I’d love to parallelize and pass data in memory without serialization. If I can’t have both, I much prefer the latter. But I still have to run on Ray for reasons external to me.

@Zoltan Eisler to turn off stats you can read up on it here.

Ha, cool! (I meant the link specifically, it’s what I needed.)