Question1: What's a good way to dynamically spawn...
# hamilton-helpm
Mattias Fornander US
04/24/2025, 4:48 PMQuestion1: What's a good way to dynamically spawn subgraphs?
Specifically, I'm trying to move away from Dataframes (logic code doesn't need column access, and need types beyond NumPy) towards lists of dataclasses as both input and output. Each "row" needs to be transformed using the same DAG, but independently. I'm looking at subdag but it doesn't fit I think. Right now I'm driving each list item independently through execute, and collecting final_vars back into a list.
Ideas?
s
Stefan Krawczyk
04/24/2025, 5:39 PMIt sounds like you want parallelizable + collect. https://hamilton.dagworks.io/en/latest/concepts/parallel-task/#using-the-parallelizable-and-collect-types
It's designed to enable you to "map" over a subgraph, e.g. Do it once for each file.
🙏 1
👀 1
Stefan Krawczyk
04/24/2025, 5:40 PMOtherwise your approach is fine. Just depends if you want to parallelize execution there at all.
m
Mattias Fornander US
04/24/2025, 10:51 PMVery cool, it worked! Any way to control the "Task" name, or plans to group a "sub-run"?
Mattias Fornander US
04/24/2025, 10:54 PMFYI duckdb didn't like being run in parallel but I somehow got around it with setting remote_executor to SynchLocalTaskExecutor:
Copy code
.enable_dynamic_execution(allow_experimental_mode=True)
.with_remote_executor(hamilton.driver.executors.SynchronousLocalTaskExecutor())👍 1
s
Stefan Krawczyk
04/25/2025, 1:07 AMNice. Log an issue for the task name? Note the navigation in the UI with parallelizable DAGs could be better.
2 Views