Hello! I have a final_var node that I want to include multiple times with different aliases for the output based on different input values, determined at config time. Roughly:

def foo(some_node: SomeNode, foo_cfg: FooConfig) -> int:
   …

# This is procedurally created with dynamic names at runtime
foo_cfgs = {"a": FooConfig(…), "b": FooConfig(…)}
dr = driver.Builder().with_config({"foo_cfgs": foo_cfgs}).build()
print(dr.execute(final_vars=[*foo_cfgs.keys()]))
#=> {"a":42, "b":17}

I have a working approach (details in thread) using

@resolve

and

@parameterize

, but I'm wondering if there's a better way to do this using pipelines or subdags or something. Basically: What's the Hamilton way (or ways) to accomplish the equivalent of this procedural code:

for name, setup in some_input.items():
    final_result[name] = some_node(…other_attrs_from_dag…, some_attr=setup)

If I understood ## What you want 1. generating

key: value

mapping outside of hamilton; let's call it it

config

2. passing

config

to the DAG via

.with_config()

3. this

config

determines what

foo

nodes are created (

foo__bar

,

foo__baz

) 4. you want to execute the DAG to get

foo__bar

,

foo__baz

, and all other

foo__

derivatives configured ## What you're currently doing • using

@resolve

to parameterization of

foo

at config time • using the config you pass to

@resolve

in your

Driver.execute()

statement # My question This seems like a working solution, albeit complex. What's the friction point you're looking to solve? In my experience,

@resolve

adds a lot of complexity and I never had to reach for it. It's typically useful for platform engineers that define reusable templates for workflows. It's common in feature engineering workflows that use Hamilton dataframe result builders for instance ## "What's the Hamilton way" > What's the Hamilton way (or ways) to accomplish the equivalent of this procedural code: >

for name, setup in some_input.items():
>     final_result[name] = some_node(…other_attrs_from_dag…, some_attr=setup)

It's not obvious what "feature" is useful to you in this for loop. Regarding Hamilton's design, the idea "you know exactly what nodes exist and how they depend on each other before `.execute()`" is at the core;

@resolve

is the main escape hatch. ## "parameterize vs for loop" I have code that generates burgers:

regular

,

cheese

and

vege

. ### config + for loop (what I would do This has a single

burger

node that dispatches implementations

def burger(ingredients: list[str]) -> Burger:
   """Builds a burger based on ingredients"""
   # could have if/else to have multiple implementations
   # burger could have `recipe_name" as kwarg

#...
recipes = {
  "regular": [...],
  "cheese": [...],
  "vege": [...],
}

dr = Builder().with_modules(...).build()

burgers = {}
for name, ingredients in recipes.items():
   results = dr.execute(["burger"], inputs={"ingredients": ingredients})
   burgers[name] = results["burger"]

You can even wrap this inside a function

def make_burger(recipe_name: str) -> Burger:
   dr = Builder().with_modules(...).build()
   recipes = {
    "regular": [...],
    "cheese": [...],
    "vege": [...],
   }
   results = dr.execute(["burger"], inputs={"ingredients": reciptes["recipe_name"]})
   return results["burger"]

### parameterize + resolve This approach creates multiple nodes

regular

cheese

,

vege

, at you can reference by name at execution. It could make things clearer or not.

@resolve_at_config(
  decorate_with=...
)
def burger_base(incredients) -> Burger:
  """Builds a burger based on ingredients"""
  # note, this is likely the same function as before

# ...
recipes = {
  "regular": [...],
  "cheese": [...],
  "vege": [...],
}

dr = Builder().with_config(**recipes).with_modules(...).build()
results = dr.execute(["cheese"], inputs={"ingredients": recipe["cheese"])
burger = results["cheese]

Wrapped inside a function, it looks the same to a user, but probably more complex to maintain

def make_burger(recipe_name: str) -> Burger:
  recipes = {
    "regular": [...],
    "cheese": [...],
    "vege": [...],
  }

  dr = Builder().with_config(**recipes).with_modules(...).build()
  results = dr.execute(["cheese"], inputs={"ingredients": recipe["cheese"])
  return results["cheese"]

Thank you. Much of this makes sense, and helps me evaluate the hackiness-or-not of my approaches. For context, there are many other nodes getting called to produce the penultimate SomeNode that's an input to the function, with complex input along the way. As a result—for my new-to-Hamilton self—it feels like calling

.execute()

N times for the outputs will be a performance problem. The execute call actually has

.execute(final_vars=[*all_the_other_outputs_requested, *foo_cfgs.keys()])

, which is to say other things are also being dynamically computed on the penultimate SomeNode for the result. I appreciate the reinforcement that "clarity of execution path and implementation" is a non-trivial benefit that Hamilton provides. The @resolve/@parameterize approach, which creates node names dynamically, seems far less good (on that metric) than my wrapper approach.

I think we are on the same page. There may or may not be a performance hit for calling

.execute()

in a loop. For good performance, you can do

dr.execute(["burger", "some_expensive_step"])

, and pass

"some_expensive_step"

as

dr.execute(..., overrides={"some_expensive_step": previous_results["some_expensive_step"]})

OR use caching. I personally prefer your wrapper approach. I think Hamilton is very powerful for "platform / data teams", but you can wrap its features inside simpler functions and classes when shipping features to your colleagues. This way, don't even need to learn anything about Hamilton. I believe that's the original pattern that Elijah & Stefan employed when creating Hamilton at Stitch Fix

Thank you. I'd not considered executing the graph part way. Very interesting!

Thanks for contributing to the community 😁

Enormous feature creep outside the scope of this discussion, so vague that it doesn't really warrant a response, but FYI: I would probably appreciate it if Hamilton's docs had a page that was roughly: "Thinking in a pull vs. push way with DAGs requires a bit of a mental re-alignment. Here are some common, general procedural coding patterns, and the way(s) to best implement them in Hamilton." Which would, of course, include exactly my case here. 😄

> "Thinking in a pull vs. push way with DAGs requires a bit of a mental re-alignment. Here are some common, general procedural coding patterns, and the way(s) to best implement them in Hamilton." We're a fully community driven project now. Happy to take contributions where we can get them 😉 🙏

I do love that, and I do want to give back. (For example, I'm planning a PR to fix a confusing typo in the docs.) However, the sort of document I'm envisioning requires someone fully steeped in (a) the experience of using Hamilton, and (b) the full knowledge of what all the pieces really do (e.g. pipelines and subdags).