How to map a dag onto a sequence? This is admittedly a bit of an outlandish question, but FWIW. Suppose I have a very simple dag

def y(x: str) -> str:
  ...

It defines a transform from the input node

x

to the output node

y

. Now I want to apply the transform to a Iterable of strings

xs: Iterable[str]

. What are my options? One approach is to build a driver and simply loop over

xs

in an external piece of code and execute the driver on each item. This has a drawback that the input

xs

and the output

ys

are not parts of the dag. Another approach is to pass iterators to the nodes

def y(x: Iterable[str]) -> Iterator[str]:
  ...

but this fuses iteration with the transform logic and is generally not a good practice. That is I want to be able to code up and debug a transform and then map it onto a sequence. Any other options?

I think what you’re describing is exactly the “parallelizable” feature. Note it doesn’t have to be run in parallel (it can be run in sequence), but it allows you to run a subdag over an iterabke and join the results. https://hamilton.dagworks.io/en/latest/concepts/parallel-task/ Otherwise you’ve correctly identified the other possibilities — but I think this is what you want!