This message was deleted.

Assuming you mean running all in one process — a common use-case is to break Hamilton DAGs that answer differnt questions across multiple tasks. So, “driver chaining” (wiring one DAG from the output of another) is something we want to do — I think this is a valuable case/is on our roadmap. That said, I think that we have a few good solutions for the time-bing: 1. Placing your graph in modules, and combining modules dynamically using a driver. They can often depend on each other, although you want your modules to be a DAG as well as your functions. The cool thing is you can use modules for (a) different discrete DAG components and (b) different implementations of subcomponents of DAGs, allowing a pretty flexible combination. 2. Calling one driver then passing the results of that (and other) to another. While we’d like to officially sum/wire together two drivers, this is a perfectly reasonable way of doing it. You lose lineage, but gain flexibility. 3.

@subdag

allows you to utilize one subcomponent (set of functions, modules, etc…) in another part of the DAG. While this isn’t as dynamic as you might want, its good if you want it set in stone You can also use overrides/set your variables you query to just get specific components, and have it all part of one DAG. My guess is modules is what you need, but you run into the problem of global namespace, which subdag (and eventually driver chaining) should help with. This is a pretty broad overview — any chance you have a quick pseudocode example to illustrate what you’re trying to do? Might help ground it more.

Yeah, so from that menu I agree modules is probably the best way to go — maybe modules with some sort of base driver-wrapper that gets subclassed to cater for different types of input (groupings of transforms) and config validation. Plugging one driver into another would work but seems clunky in a context you have more than one class of data in the same set of inputs. A subdag would be overly restrictive for this use-case.

For modules the route, I think you should be able think about it visually. Given a DAG visualization can we cut it up in a way, that allows you to segment parts of the DAG for each context. So that way it’s easy to run just that part, but also allows you to run it altogether. The nice part here, is that with @config.when, you also have some flexibility to adjust the shape based on “config” in addition to partitioning via modules…