https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Is there an option for @load_from to use a default...
# hamilton-help
v
Is there an option for @load_from to use a default value, in case a file / source doesn't yet exist?
đź‘€ 1
e
To clarify, think you’re looking for something like a fallback path, right? E.G. if some path doesn’t exist, go to a backup? Otherwise use the parameter value? If you’re just looking for default values if you don’t want to provide them, you can specify it in the dataclasses that define it — see a list + locations in code here: https://hamilton.dagworks.io/en/latest/reference/io/available-data-adapters/#data-loaders. Otherwise this is doable but not super ergonomic if I understand correctly — you can refer to another node in the 
load_from
, and use that node to give you the path.
Copy code
@load_from.csv(
    path=source("path_to_load")
)
def foo(data: pd.DataFrame) -> pd.DataFrame:
    ...

def path_to_load() -> str:
    return "/foo/bar" if os.path.exists("/foo/bar") else ...
You can also define a custom loader class which will encode this…
v
I'm loading and saving a function's parameter "test_textwidget_value" and return value from/to the same source. Of course, to start, there is nothing in that file and I'm getting an exception. I'd like an empty value to be returned (here: empty string) in that case, and otherwise the contents of that file. 
@load_from.pickle(path=mysource, inject_="test_textwidget_value")
@save_to.pickle(path=mysource, output_name_="textwidget_save")
def test_textwidget(test_textwidget_value: str='') -> str:
...
e
Ahh, so yeah, this currently isn’t supported — you’d want to effectively override the pickle loader to do that/get it to work with your desired behavior. Will send some code shortly, but at a high levels: 1. The pickle loader currently fails if nothing exists 2. You want it to return an empty string (may I suggest 
None
? Might be a bit cleaner) 3. you’ll need to change the pickle loader class, or add a custom widget loader (load_from.widget) 4. You can do so by registering a loader for pickle (it’ll overwrite the default), and looking at the pickle loader, or by creating a widget loader 5. You should
Will get code soon
v
thank you Elijah, makes sense
e
Hey — haven’t forgot, will add code today 🙂
Hey! Sorry for the delay. Here’s what a attempt-with-fallback would look like — haven’t tested it out yet but it or something similar should work. This returns an empty string if it does not exist, I’d consider 
None
as well:
Copy code
@dataclasses.dataclass
class PickleLoader(DataLoader):
    path: str

    @classmethod
    def applicable_types(cls) -> Collection[Type]:
        return [object, Any]

    @classmethod
    def name(cls) -> str:
        return "pickle"

    def load_data(self, type_: Type[object]) -> Tuple[Optional[object], Dict[str, Any]]:
        if not os.path.exists(self.path):
            return None 
        with open(self.path, "rb") as f:
            return pickle.load(f), get_file_metadata(self.path)

# make sure you call this code at some point, after importing Hamilton
from hamilton.registry import register_adapter
register_adapter(PickleLoader)
Basically it just overrides the default pickle loader
v
amazing! Thank you.
I've done it by subclassing and with exception handling, but your example may be more appropriate.
e
Yeah! That’ll work as well.
Open in Slack
PreviousNext
Powered by