https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
This message was deleted.
# hamilton-help
s
This message was deleted.
šŸ‘€ 1
s
Thanks for the question! I have a meeting and will get back to you shortly.
You have a few options.
You parameterize the DAG based on the config ā€” so that we construct a DAG for each fixed parameterization.
So you can have
Copy code
@config.when(mean_type='A')
def moving_average__A(input1: ... ) -> ...:
    # logic for A version
    ...

@config.when(mean_type='B')
def moving_average__B(input1: ... ) -> ...:
    # logic for B version
    ...
That will switch out and run different logic ā€” you can hard code values here if you like and itā€™ll be close to the transforms.
Since Hamilton is python object agnostic ā€” if youā€™re happy coupling your code to that class object, declare it as input and use it directly in the function:
Copy code
@config.when(mean_type='special')
def moving_average__special(custom_object: ITS_TYPE, input1: ...) -> ...:
   # logic to use the custom object... e.g.
   return custom_object.compute(input1)
Then at driver time you just need to pass in the right ā€œDAG configā€, and then any inputs that are required to run the DAG.
In terms of
Iā€™d like to keep track of all of these parameters in some python (data)class/attrs object.
you have options ā€” depends on how you want to manage that code. i.e. with the transforms themselves (as alluded to above), with the driver code, or outside of hamilton code entirely.
So the other options are something like:
Copy code
def moving_average(config_required: dict) -> ...:
   # use a dict of things to parameterize this function's logic 
   ...
and the driver code would be
Copy code
# for first DF
config = {
  'config_required': {...} # or load from some other place
  'input': df_1
}
dr = driver.Driver(config, modules)
df = dr.execute([...])

# for second DF
config = {
  'config_required': {...} # or load from some other place
  'input': df_2
}
dr = driver.Driver(config, modules)
df = dr.execute([...])
does that make sense @Zouhair Mahboubi?
z
thanks - I think I get it. What I was missing is that config can contain all of the set of ā€œinputsā€ Couple of followups: 1. Do the objects in the Driver config object (
config_required
in this example) need to be json serialazable (dicts/etc.) or can they be any python object (e.g. a callback, a class, etc.) 2. can these other parameters/objects be in inputs vs in config? a. and whatā€™s the difference between putting inputs in the config vs passing it to driver.execute as shown for example here ?
e
Yep ā€” its really open-ended. 1. Any python object is fine (although serializable will probably make your life easier at some point down the lineā€¦). If youā€™re passing in a callback or class Iā€™d highly recommend using the type of config-based delegatation @Stefan Krawczyk recommended and baking it in the dataflow itself. E.G. if instead of passing in a function you embed that into the DAG and choose it depending on a parameter, your life will get a lot easier. 2. Yep! Either one is fine. The difference between config and input is a little subtle. config allows you to use it at DAG construction (E.G. with 
config.when
). So it can actually shape the DAG. Inputs (as well as config) are passed in during runtime ā€” so you can actually declare a dependency on an it. If itā€™s not passed in, execution will error out!
So:
Copy code
# can be run with bar supplied in inputs *or* config
def foo(bar: int) -> int:
    ...

# bar has to be supplied in "config"
@config.when(bar=1)
def foo__1() -> int:
    ...
Note thereā€™s the notion of overrides (not 
config
or 
inputs
but a third, more mysterious thing šŸ˜†) that enables you to short-circuit node execution and replace a node. But I digressā€¦
z
awesome - thanks. I think thatā€™s likely enough to get me started with a prototype and see what I break šŸ˜›
e
Awesome! Happy to answer more qs as they come up :)
Open in Slack
PreviousNext
Powered by