Victor Bouzas
08/20/2024, 9:32 AMdef step1(arg: int):
return some_func(arg)
def step2(arg: int):
return some_other_func(arg)
def final_step(step1: int):
return step1
Is it possible on the use case above to define two DAGs one that goes step1 -> final_step and other step2 -> final_step, without rewriting the final step two times or using multiple arguments to it (imagine instead of two I have 10+ steps) ?Yaser Martínez Palenzuela
08/20/2024, 11:36 AMThierry Jean
08/20/2024, 2:19 PMfinal_step() in all of them without duplicating code.
## Defining multiple DAGS
For managing several DAGs, it's usually most convenient to use separate .py files (e.g., dag_foo.py and dag_bar.py).
Define your first DAG dag_foo
# dag_foo.py
def raw_data() -> pd.DataFrame:
"""This loads country data"
def cleaned_data(raw_data: pd.DataFrame) -> pd.DataFrame:
"""Remove rows with null values"""
Define the second DAG dag_bar. It can have a totally different structure, but ends with cleaned_data
# dag_bar.py
def api_data() -> list:
"""Query city data"
def cities(api_data: list) -> pd.Dataframe:
"""Create a dataframe from API JSON response"""
def cleaned_data(cities: pd.DataFrame) -> pd.DataFrame:
"""City data cleaned for analysis"""
## Defining common steps
Define the common nodes in common_steps
# common_steps.py
def final_step(cleaned_data: pd.DataFrame) -> dict:
"""Compute some statistics"""
## Building from multiple modules
Build the Driver with dag_foo + common_steps or dag_bar + common_steps
# run.py
from hamilton import driver
import dag_foo
import common_steps
dr = driver.Builder().with_modules(dag_foo, common_steps).build()
Now, I understand that you would like "two versions" of final_steps with different relationships. The most "hamiltonic" way would be to use @config.when as Yaser suggested! It allows to have two functions with the same name in the same module and then select which one to load.
# common_steps.py
from hamilton.function_modifiers import config
@config.when(dag_version="foo")
def final_step__foo(cleaned_data: pd.DataFrame):
"""Compute some statistics"""
@config.when(dag_version="bar")
def final_step__bar(cleaned_data: pd.DataFrame, external_input: str):
"""compute statistics and prepare report"""
Then the Builder().with_config() allows to specify the keys dag_version="foo" or dag_version="bar" to load either one
# run.py
from hamilton import driver
import dag_foo
import common_steps
dr = (
driver.Builder()
.with_modules(dag_foo, common_steps)
.with_config(dict(dag_version="foo")
.build()
)
I'll stop here and wait to hear your thoughts! I'll be available to help you figure this out !Yaser Martínez Palenzuela
08/20/2024, 3:11 PMVictor Bouzas
08/20/2024, 8:54 PMfinal_step__bar and final_step__foo in my scenario is the input, not the function logic itself. I’m dealing with 10+ different processes (combinations of steps), some of which share common steps, while others are unique. Duplicating code just to adjust connections doesn’t seem like the best approach. Am I thinking about this the right way?
@Stefan Krawczyk, thanks for the video you shared! After watching it, I’m considering breaking down the processes into multiple DAGs and then combining them later. However, I worry that this cam make things a lot more complex and difficult to visualize, so I’d prefer to represent each process with a single DAG.
Is there any way I could manually assemble the data flow rather than relying solely on the parsing of function dependencies? I hope that makes sense, and I’d love to hear any other suggestions or alternative approaches to tackle this use case! 😊Stefan Krawczyk
08/20/2024, 9:31 PM_ prefix aren’t included in the DAG):
def _common_logic(...) -> ...
# common logic
def asset1( ... ) -> ... :
return _common_logic(...)
def asset2( ... ) -> ... :
return _common_logic(...)
there’s also a few other ways to reuse/parameterize/inject things. See the pandas split apply combine example to see a few in action. What you use depends on the context & dynamism & obviously what you want to accomplish.
regarding:
> Is there any way I could manually assemble the data flow rather than relying solely on the parsing of function dependencies?
What is the friction point you’re encountering? Can you provide some more context here or an example?Victor Bouzas
08/21/2024, 7:48 AMconfig, my understanding is that I would have to copy the function with different config decorators, as below. That would be a friction to do considering I don't have only this common node with changing connections, but multiple combinations.
@config.when(dag_version="dag_1")
def B__fromA(A: int) -> int:
return A + 2
@config.when(dag_version="dag_2")
def B__fromD(D: int) -> int:
return D + 2
@config.when(dag_version="dag3")
def B__fromE(E: int) -> int:
return E + 2
The ideal scenario I would have a single function but defining only the different connections (either by assembling the dataflow manually or using another config for args maybe):
@config_args.when(dag_version={
"dag_1": {"input": A},"dag_2": {"input": D},"dag_1": {"input": E}
})
def B(input: int) -> int:
return input + 2
I hope I was clear, but feel free to ask for more clarifications or challenge the approach I am using.Viktor
08/21/2024, 1:30 PMVictor Bouzas
08/21/2024, 4:20 PMStefan Krawczyk
08/21/2024, 5:05 PMVictor Bouzas
08/21/2024, 5:29 PMStefan Krawczyk
08/21/2024, 5:48 PMdag_config = {
"dag_name": "dag_1",
"hamilton.enable_power_user_mode": True
}
(I’m using jupyter cell magic here):
%%cell_to_module option_2 --display --config dag_config
from hamilton.function_modifiers import resolve_from_config, inject, source, value, config, parameterize, group
B_options = {"dag_1": "A","dag_2": "D","dag_3": "E"}
@resolve_from_config(
decorate_with=lambda dag_name: inject(
input1=source(B_options[dag_name])
)
)
def B(input1: int) -> int:
return input1
Would give:Stefan Krawczyk
08/21/2024, 5:50 PMStefan Krawczyk
08/21/2024, 5:52 PMVictor Bouzas
08/21/2024, 6:21 PM