So the hamilton driver should be serializable. So you could instantiate one and then distribute it.

Otherwise it sounds like you want each of the parallel drivers to execute in unison? Versus independently?

I have a few more thoughts — but have to go AFK. Will respond in a bit.

So two approaches I can think of if you need them to be in sync: Decentralized 1. Idea is that you run things in parallel, but build a way to “sync” progress between the processes. 2. Approach: need a datastore to sync against that can do atomic updates. 3. Build an adapter that (see lifecycle adapters, blog) that syncs progress to a datastore and only continues if all X processes have completed. E.g. you provide the number of processes & some ID for the run and each will update and then wait for all to complete before continuing… 4. Caveat: order of execution matters here - I think it’s stable, but don’t quote me on that (or it could be something to fix by adding a sort somewhere). 5. The Hamilton UI should be able to work here - it would track each individual driver run. Centralized: 1. So if you need to parameterize over something, I think this is a bit trickier - is that something you need? 2. If you don’t need to parameterize then I think turning on the new execution mode and building your own executor could work — the executor would be in charge of make N copies of the function run somewhere before moving onto the next one; you’d then need to co-ordinate passing in the right data for each one. The Hamilton UI here might need some help in figuring out what to track properly… 3. what you kind of want is what parallelizeable + collect does, but instead of a task being a subgraph, it’s a BFS walk of the graph and parallelizing each node like you want… @Elijah Ben Izzy might have some more thoughts — but yeah if you want to go centralized it’s some amount of code to be augmented to make it work nicely….