https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
This message was deleted.
# hamilton-help
s
This message was deleted.
e
Welcome @Jan Hurst! Sorry I didn’t get to this last night.
OK, so you’re 99% of the way there. For what you want to do, I think you’re missing two (very small) pieces: 1. The function to 
@resolve
references the config items by name. So, what you want is (I think)
Copy code
@resolve(
    when=ResolveAt.CONFIG_AVAILABLE,
    decorate_with=lambda names: parameterize_values("name", {(name, doc_string): name for name in names})
)
The way to read this is that, when the config is available, we get the 
names
field from it and utilize it to build the 
paramterize_values
decorator, which then gets applied to 
create_obj
. 2. The variables to 
driver
get called as a list 
dr.execute(['name1')
I’ve gotten your example to work (using 
@parameterize
instead but its basically the same. Code should be able to run!
Copy code
from hamilton.function_modifiers import resolve, ResolveAt, parameterize, value

from hamilton.ad_hoc_utils import create_temporary_module
from hamilton import driver
from hamilton.base import SimplePythonGraphAdapter, DictResult

import dataclasses

@dataclasses.dataclass
class MyObject:
    name: str

@resolve(
    when=ResolveAt.CONFIG_AVAILABLE,
    decorate_with=lambda names: parameterize(
        **{name : {"name" : value(name)} for name in names}
    )
)
def create_obj(name: str) -> MyObject:
    return MyObject(name)


mod = create_temporary_module(create_obj)
adapter = SimplePythonGraphAdapter(DictResult())
dr = driver.Driver(
    {
         "hamilton.enable_power_user_mode" : True,
         "names" : ["name1", "name2"]
    },
    mod,
    adapter=SimplePythonGraphAdapter(DictResult())
)

print(dr.execute(["name1"]))
Now, say you want to run multiple steps for each of these “names” — this is where you can look into 
subdag
. 
subdag
+ 
@parameterize_subdag
allow just that: • https://hamilton.readthedocs.io/en/latest/reference/api-reference/decorators.html#subdag • https://hamilton.readthedocs.io/en/latest/reference/api-reference/decorators.html#parameterized-subdag That said, you’ve jumpted straight to power-user mode, so I’d be careful. Other options are: 1. Have the config hardcoded if you dont want to change it that much 2. Have a for-loop inside your function
👍 1
j
thanks! i had started sketching around both 
@resolve
and 
@parameterized_subdag
and got some things up and running. the bit i got stuck on was figuring out how to "ask" for a specific output in a subdag, i.e. i didn't understand the 
name1.subdag_step
output naming. this was compounded a bit by not being able to get the visualization stuff to work very well im also now exploring if i can do something outside of the DAG to setup a better config. one of the important things I am looking for is to have very atomic nodes so that i can farm out to Dask (which incidentally was a snap to use)
e
Oh, yeah, this is actually pretty simple 🙂 (not sure if you’ve figured it out but I’ll put it here just in case): Take the example here:
Copy code
@subdag(
    *modules,
    inputs={...},
    config={...}
)
def some_fn(subdag_output: ...) -> ...:
    return do_something_with(subdag_output)
the subdag produced will be namespaced by 
some_fn
and contain a final element called 
subdag_output
. That will then be passed to 
some_fn
which will be a node in the DAG. So, it’ll be: 
… -> some_fn.subdag_output -> some_fn -> …
Makes sense? Would love to make this clearer in the documentation if possible.
Re: setting up a config well, I’ve seen some people use Hamilton to do that as well, but my general tip is make it as simple as possible (few items, each one with specific meaning), Think of it as the interface to what you’re building — you want as few, meaningful, arguments as possible.
Glad dask was easy!
Open in Slack
PreviousNext
Powered by