Is this possible to parameterise the ` save to pickle` decor Hamilton Open Source #hamilton-help

Is this possible to parameterise the `@save_to.pic...

Konstantin Sofeikov

12/19/2024, 5:41 PM

Is this possible to parameterise the

@save_to.pickle

decorator in this context so the result of each node after parameterise is saved to a different file?

Copy code

@parameterize(
    team_nodes_one_hot=dict(target_df=source("team_nodes"), node_type=value("team")),
    player_nodes_one_hot=dict(target_df=source("player_nodes"), node_type=value("player"))
)
@config.when(stage="training")
@save_to.pickle(path=value('encoders.pkl'))
def one_hot_categorical_encoding__training(
    target_df: pl.DataFrame, data_config: DataConfig, node_type: str
) -> dict[str, dict[str, OneHotEncoder]]:

Thanks!

Stefan Krawczyk

12/19/2024, 6:26 PM

So this stacking won’t work due to the order of how we resolve decorators — specifically the @save_to can’t be used here. (CC @Jernej Frank @Elijah Ben Izzy - related to decorator convo on Tuesday).

🤔 1

Stefan Krawczyk

12/19/2024, 6:28 PM

But you can parameterize it this way:

Copy code

# my_reusable_dag.py
import pandas as pd
from typing import Dict, Any
from hamilton.function_modifiers import parameterize, source, value, save_to, config, subdag

@save_to.pickle(path=source('encoder_output_path'))
@config.when(stage="training")
def one_hot_categorical_encoding__training(
    target_df: pd.DataFrame, data_config: Dict, node_type: str
) -> dict[str, dict[str, Any]]:
    pass

You could then do manual subdag incantations:

Copy code

import pandas as pd
from typing import Dict, Any
from hamilton.function_modifiers import parameterize, source, value, save_to, config, subdag
import my_reusable_dag

@subdag(
  my_reusable_dag,
  config={"stage":"training"},
  inputs=dict(target_df=source("team_nodes"), node_type=value("team"), encoder_output_path=source("encoder_output_path"))
)
def team_nodes_one_hot(one_hot_categorical_encoding: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
    return one_hot_categorical_encoding

or use the parameterized_subdag decorator:

Copy code

import pandas as pd
from typing import Dict, Any
from hamilton.function_modifiers import parameterize, source, value, save_to, config, subdag, parameterized_subdag
import my_reusable_dag

@parameterized_subdag(
  my_reusable_dag,
  team_nodes_one_hot=dict(config={"stage":"training"},
                          inputs=dict(target_df=source("team_nodes"), node_type=value("team"), encoder_output_path=source("encoder_output_path"))),
  player_nodes_one_hot=dict(config={"stage":"training"},
                          inputs=dict(target_df=source("player_nodes"), node_type=value("player"), encoder_output_path=source("encoder_output_path")))  
)
def encoder(one_hot_categorical_encoding: dict[str, dict[str, Any]]) -> dict[str, dict[str, Any]]:
    return one_hot_categorical_encoding

We do suggest verbose ways are better than terse ones. It’s less intimidating to read.

Stefan Krawczyk

12/19/2024, 6:31 PM

Just to mention - to make it more parameterizable you could also add resolve_with_config that could dynamically create the decorators based on the passed in global configuration when the DAG is built. Don’t worry if this is too much, it’s quite easy to 🤯 with the levels of parameterization you can achieve with Hamilton.

Konstantin Sofeikov

12/19/2024, 6:32 PM

Yeah, I was afraid it was gonna end up being a dag parameterisation lol. Appreciate your help ,thanks!

Elijah Ben Izzy

12/19/2024, 7:26 PM

Yeah it’s going to be difficult to do with the vanilla tooling as Stefan said — largely cause there’s a lot of things that need to be specified to make it work. You can wrap that in your own decorator if you want to make it more readable (maybe even contribute that back?) — can show code later if that’s interesting

2 Views

Open in Slack

Previous Next