Hey folks! Just discovered Hamilton a few days ago...
# hamilton-help
v
Hey folks! Just discovered Hamilton a few days ago and was exactly what I was looking for to my use case (lightweight DAG builder and executor). However, my understanding is that the DAG is always built dynamically from the functions based on arguments.Is it possible to define the connection between nodes explicitly and not build the DAG automatically ? I ask this because I have one use case that I need to reuse some nodes between multiple DAGs but with different connection between them. Is there another way to do this easily ? Example:
Copy code
def step1(arg: int):
   return some_func(arg)

def step2(arg: int):
   return some_other_func(arg)

def final_step(step1: int):
   return step1
Is it possible on the use case above to define two DAGs one that goes
step1 -> final_step
and other
step2 -> final_step
, without rewriting the final step two times or using multiple arguments to it (imagine instead of two I have 10+ steps) ?
t
Hi @Victor Bouzas 👋 Happy you're finding Hamilton useful! I'm curious about what kind of problem it helps you solve! My understanding is that you have several DAGs that do different things, but you'd like to reuse the node
final_step()
in all of them without duplicating code. ## Defining multiple DAGS For managing several DAGs, it's usually most convenient to use separate
.py
files (e.g.,
dag_foo.py
and
dag_bar.py
). Define your first DAG
dag_foo
Copy code
# dag_foo.py
def raw_data() -> pd.DataFrame:
  """This loads country data"

def cleaned_data(raw_data: pd.DataFrame) -> pd.DataFrame:
   """Remove rows with null values"""
Define the second DAG
dag_bar
. It can have a totally different structure, but ends with
cleaned_data
Copy code
# dag_bar.py
def api_data() -> list:
  """Query city data"

def cities(api_data: list) -> pd.Dataframe:
  """Create a dataframe from API JSON response"""

def cleaned_data(cities: pd.DataFrame) -> pd.DataFrame:
   """City data cleaned for analysis"""
## Defining common steps Define the common nodes in
common_steps
Copy code
# common_steps.py
def final_step(cleaned_data: pd.DataFrame) -> dict:
   """Compute some statistics"""
## Building from multiple modules Build the Driver with
dag_foo + common_steps
or
dag_bar + common_steps
Copy code
# run.py
from hamilton import driver
import dag_foo
import common_steps

dr = driver.Builder().with_modules(dag_foo, common_steps).build()
Now, I understand that you would like "two versions" of
final_steps
with different relationships. The most "hamiltonic" way would be to use
@config.when
as Yaser suggested! It allows to have two functions with the same name in the same module and then select which one to load.
Copy code
# common_steps.py
from hamilton.function_modifiers import config

@config.when(dag_version="foo")
def final_step__foo(cleaned_data: pd.DataFrame):
  """Compute some statistics"""

@config.when(dag_version="bar")
def final_step__bar(cleaned_data: pd.DataFrame, external_input: str):
  """compute statistics and prepare report"""
Then the
Builder().with_config()
allows to specify the keys
dag_version="foo"
or
dag_version="bar"
to load either one
Copy code
# run.py
from hamilton import driver
import dag_foo
import common_steps

dr = (
  driver.Builder()
  .with_modules(dag_foo, common_steps)
  .with_config(dict(dag_version="foo")  
  .build()
)
I'll stop here and wait to hear your thoughts! I'll be available to help you figure this out !
👍 1
y
great response too 🙂
👍 1
s
Note: there are other approaches that could be used, but it depends on your goals and scale, etc. I discuss some of these

here▾

(it starts around 28:30).
👍 2
v
Hey everyone, Thank you so much for your insights! Apologies for the delay in getting back to you—I’ve been testing and reviewing everything. @Thierry Jean, I really appreciate your suggestion on separating the files! 😊 My use case is a bit unique. I'm modeling multiple manufacturing processes as DAGs (since they essentially represent sequences of nodes) and I'm looking for a lightweight way to build and run these DAGs while keeping track of intermediate results. In the example you provided with configs, I’m concerned that it might lead to duplicating code multiple times, just with a different decorator. The only difference between
final_step__bar
and
final_step__foo
in my scenario is the input, not the function logic itself. I’m dealing with 10+ different processes (combinations of steps), some of which share common steps, while others are unique. Duplicating code just to adjust connections doesn’t seem like the best approach. Am I thinking about this the right way? @Stefan Krawczyk, thanks for the video you shared! After watching it, I’m considering breaking down the processes into multiple DAGs and then combining them later. However, I worry that this cam make things a lot more complex and difficult to visualize, so I’d prefer to represent each process with a single DAG. Is there any way I could manually assemble the data flow rather than relying solely on the parsing of function dependencies? I hope that makes sense, and I’d love to hear any other suggestions or alternative approaches to tackle this use case! 😊
s
@Victor Bouzas we had an interesting talk today at the meet-up that could be relevant — once the recording is available will send that over. On the visualization front, the hamilton UI does have some affordances there to help there. e.g. you can pan and zoom more easily than graphviz. In case it’s useful, you can create helper functions (functions with
_
prefix aren’t included in the DAG):
Copy code
def _common_logic(...) -> ...
   # common logic

def asset1( ... ) -> ... :
  return _common_logic(...)

def asset2( ... ) -> ... :
  return _common_logic(...)
there’s also a few other ways to reuse/parameterize/inject things. See the pandas split apply combine example to see a few in action. What you use depends on the context & dynamism & obviously what you want to accomplish. regarding: > Is there any way I could manually assemble the data flow rather than relying solely on the parsing of function dependencies? What is the friction point you’re encountering? Can you provide some more context here or an example?
v
Thanks, I will take a look at this later today. My scenario is that I have many nodes (functions), but they can be combined in different ways to form multiple DAGs. The main difference between nodes in the different DAGs is the connections and some external parameters that I can provide, not the function itself. For example, in the 3 DAGs below, I would like a way to reuse node B without rewriting the function to use a different input. For example, if I used
config
, my understanding is that I would have to copy the function with different config decorators, as below. That would be a friction to do considering I don't have only this common node with changing connections, but multiple combinations.
Copy code
@config.when(dag_version="dag_1")
def B__fromA(A: int) -> int:
    return A + 2

@config.when(dag_version="dag_2")
def B__fromD(D: int) -> int:
    return D + 2

@config.when(dag_version="dag3")
def B__fromE(E: int) -> int:
    return E + 2
The ideal scenario I would have a single function but defining only the different connections (either by assembling the dataflow manually or using another config for args maybe):
Copy code
@config_args.when(dag_version={
"dag_1": {"input": A},"dag_2": {"input": D},"dag_1": {"input": E}
})
def B(input: int) -> int:
    return input + 2
I hope I was clear, but feel free to ask for more clarifications or challenge the approach I am using.
v
@Victor Bouzas are you thinking DAGs of "assets" or of "tasks"? You can easily – just as with any library e.g. like pandas – have reusable functions in their own non-hamilton module/s and just use them within the asset definition functions in the modules, that actually consitute the different DAGs.
v
I’m referring of DAGs as a sequence of nodes. I understand your suggestion, but wouldn’t I have to create a new custom “Hamilton” function every time I want to change the connection of this node in this option as well ?
s
@Victor Bouzas question — to confirm, you don’t want one large DAG, instead just a way to have 3 (in your example) different DAGs that share logic where appropriate? I believe yes, so correct me if I’m wrong. Can sketch some code in a few for you.
👍 2
v
Yes, that’s it @Stefan Krawczyk . Ok, Thanks a lot!
s
@Victor Bouzas something like:
Copy code
dag_config = {
   "dag_name": "dag_1", 
   "hamilton.enable_power_user_mode": True
}
(I’m using jupyter cell magic here):
Copy code
%%cell_to_module option_2 --display --config dag_config

from hamilton.function_modifiers import resolve_from_config, inject, source, value, config, parameterize, group

B_options = {"dag_1": "A","dag_2": "D","dag_3": "E"}

@resolve_from_config(
    decorate_with=lambda dag_name: inject(
        input1=source(B_options[dag_name])
    )
)
def B(input1: int) -> int:
    return input1
Would give:
okay I think this incarnation above is closest to what you want.
note: the decorator could be simplified by wrapping it in another decorator if you end up putting it in a lot of places…
v
Ohhh, that is great, tested here and it covers this use case perfectly. Thanks a lot for the support! 🙂
🙌 1
🫡 1