https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Parameterization I am developing this dag for (rig...
# hamilton-help
v
Parameterization I am developing this dag for (right side of image) • reading multiline strings from pickle files • cleaning them • and amending them Currently, for all three pathways, the same file is being read, for • 
url_regexes
• 
urls
• 
queries
I would like to parameterize the left-most input 
textwidget_location str
, which is the file name to use, for each of these three branches, so this currently one path becomes three separate ingress pipes.
What is the best way to do this?
The sequence
textwidget_location -> ... -> textwidget_load
is in a separate module.
This is the definition of the Loading module
Copy code
%%cell_to_module -m textwidget_store
from hamilton.function_modifiers import load_from, save_to, source

ttws = source("textwidget_location")

@load_from.pickle(path=ttws, inject_="textwidget_value")
def textwidget_load(textwidget_value: str='') -> str:
    '''loading a stored textarea widget'''
    return textwidget_value

@save_to.pickle(path=ttws, output_name_="textwidget_save")
def textwidget_save_preprocess(textwidget_value: str='') -> str:
    '''saving a stored textarea widget'''
    return textwidget_value
s
@Viktor how many parameterizations would there be? Is it a fixed known number? or is it something you don’t know until runtime?
For reference I explain a few approaches in

this video▾

. Otherwise if it’s dynamic and you don’t know ahead of time I would look https://hamilton.dagworks.io/en/latest/concepts/parallel-task/. Otherwise if it’s a fixed number/set of files there are ways to specify that with @parameterize / @parameterize_subdag, or just manually being verbose.
For parallel the sketch would be:
Copy code
def textwidget_location(paths: list[str]) -> Parallelizable[str]:
    for path in paths:
       yield path

@load_from.pickle(path=source("textwidget_location"), inject_="textwidget_value")
def textwidget_load(textwidget_value: str='') -> str:
    '''loading a stored textarea widget'''
    return textwidget_value

...

def collected_data(... : Collect[dict]) -> ...:
   # you need a collect to aggregate things back -- this could be at the end, or in the middle
v
Thank you! Yes, it's a fixed number. Only the three nodes • raw_urls_multiline • raw_url_regexes_multiline • raw_queries_multiline will be loading each their own file through the textwidget_store module. I'm basically looking to connect each of these to the 
textwidget_load
node in the 
textwidget_store
module (subdag?), but with their own file source request, so to say. I would be able to use pipe with the 
textwidget_load
node, if only the file source parameter wasn't on the separate 
load_from
node coming from the decorator … I wonder, how this works with Hamilton, when the parameter needs to be applied on a higher upstream node, than the dependency node. Will look into the videos.
… so, I see what you're saying. In my case, I am not looking to execute the dag with multiple inputs at once, but to connect a node of one dag to an other dag, with a specified source as context (here: "textwidget_location"), that is the pickle file name.
s
> Only the three nodes > • raw_urls_multiline > • raw_url_regexes_multiline > • raw_queries_multiline > will be loading each their own file through the textwidget_store module. > … > In my case, I am not looking to execute the dag with multiple inputs at once, but to connect a node of one dag to an other dag, with a specified source as context (here: “textwidget_location”), that is the pickle file name. Do you need to change your DAG at all then? IIUC you could just do this:
Copy code
file_1_result = dr.execute(["urls_regexes"], inputs={"textdwidget_location": file_1})

file_2_result = dr.execute(["urls_multiline"], inputs={"textdwidget_location": file_2})

file_3_result = dr.execute(["preprocessed_queries"], inputs={"textdwidget_location": file_3})
So you’d need to manage the output desired with the correct input file.
v
my bad, sorry for the misunderstanding. These three nodes will eventually be combined as inputs to a subsequent node, — so I have all three nodes in one module drawing content from pickle files via the same separate second module … I found the subdag decorator in the docs. It looks almost right. I'll try that.
This is the relevant dag
textwidget_load is a separate module
s
So to clarify, you need to pass in 3 source file inputs for it to run, is that right?
v
yes I'd like to have textwidget_load as three separate nodes – not just once like in the image, each its own source file input coming from the respective three leftmost nodes.
image.png
if textwidget_load was just one function, I'd use pipe
but the source definition is on the load_from decorator, a node further upstream ...
s
Yep - so a few ways: 1. just be verbose and name things independently with independent functions and wire the downstream to depend on them.
Copy code
@load_from.pickle(path=source("textwidget_location_1"), inject_="textwidget_value")
def textwidget_load_1(textwidget_value: str='') -> str:
    '''loading a stored textarea widget'''
    return textwidget_value

@load_from.pickle(path=source("textwidget_location_2"), inject_="textwidget_value")
def textwidget_load_2(textwidget_value: str='') -> str:
    '''loading a stored textarea widget'''
    return textwidget_value

@load_from.pickle(path=source("textwidget_location_3"), inject_="textwidget_value")
def textwidget_load_3(textwidget_value: str='') -> str:
    '''loading a stored textarea widget'''
    return textwidget_value

# then downstream you'd depend on the correct one above ^
2. You could do subdag that mirrors what I did executing the DAG three times
Copy code
@subdag(..., inputs={"textdwidget_location": source("filelocation1")})
def file_1_result(urls_regexes: str) -> ...
    return urls_regexes

@subdag(..., inputs={"textdwidget_location": source("filelocation2")})
def file_2_result(urls_multiline: str) -> ...
    return urls_multiline
...
# then you'd join these
def dimension_filters(file_1_result: str, ..) -> ...
3. You could do just a subdag over the data loading — you’d then still need to rewire the downstream functions to depend on that subdag function like in (1).
v
Thank you! Very useful.
👍 1
Open in Slack
PreviousNext
Powered by