https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
hello, is it possible to reference a `to` material...
# hamilton-help
m
hello, is it possible to reference a 
to
materialization to execute a function after it happens?
Copy code
# run.py
materializers = [to.csv(id="save_file", dependencies=["file_modify"], path="./out.csv")]
...driver build and execution

# pipeline.py
def load_saved_file(save_file, remote_path):
    fs.put(save_file, remote_path)
When I try to reference like this I get 
Duplicate node names found
Any advice?
s
What's the reason for trying to do it this way? Versus being more explicit? Just trying to get a sense of the requirements.
m
hey 👋, I'm trying to use it for a data pipeline where we do an extract from a mysql db -> apply some transformations -> save as CSV (with the materializer) -> perform a Copy command to load to Redshift DB. This last part I was expecting to execute it via a function right after the CSV materializer completed
s
I see -- also to clarify:
Copy code
def load_saved_file(save_file, remote_path):
    fs.put(save_file, remote_path)
you want 
save_file
here to be passed the value 
"./out.csv"
right? or an actual file handle?
👍 1
m
correct, based on the path I can load the correct file system
I actually missed a step, it should be as follows: sql extract -> transform -> csv output locally (via materializer) -> upload to S3 -> perform copy to DB
s
materializers = [to.csv(id="save_to_csv", dependencies=["file_modify"], path=source("save_file_path")]
...driver build and execution
# pipeline.py
def load_saved_file(save_file_path, remote_path):
fs.put(save_file, remote_path)
I think you want something more like this, no? I forget whether we allow 
source
in the materializer bit -- but it feels like that's the thing you want to connect right?
m
so, if I reference the same materializer path in the 
load_saved_file
function it would build the dependency between those 2 components (
save_to_csv
---then---> 
load_saved_file
)?
s
Ah I see your point; you are correct we want to specify the order here too.
CC @Thierry Jean if you're around.
m
yeah, one of the main selling points of hamilton , for me at least, is the ability to explain the steps and dependencies between them
s
I assumed you saw https://hamilton.dagworks.io/en/latest/concepts/materialization/ ? I don't think we have an example that shows what you want. I have to run -- but I'll try to sketch something out by EOD tomorrow. @Thierry Jean if you're around please chime in.
m
yeah, checked the docs but couldn't find any reference that could help my use case 😞
s
One more question @Mauricio how are you running it? via .materialize? or with_materializers ?
m
with_materializers
👍 1
s
@Mauricio So this works:
Copy code
from hamilton.function_modifiers import save_to, source, inject
import pandas as pd

def data_from_sql(sql: str)->list:
    return []

@save_to.csv(path=source("file_path"))
def transformed_data(data_from_sql: list) -> pd.DataFrame:
    return data_from_sql

@inject(dep=source("save.transformed_data"))
def upload_data(file_path: str, dep: dict) -> dict:
    return {}

def copy_to_db(upload_data: dict) -> dict:
    return {}
Using the 
save_to
decorator -- because it's resolved when building the graph
if you replace the 
@save_to.csv(path=source("file_path"))
with the 
to.csv
version using 
with_materializers
then yes I get the 
Duplicate node names found
. This is a constraint in the code right now -- it could be relaxed ... How dynamic do you want this to be? Since you could use 
@config.when
/ different modules to build the graph from to possibly achieve the same result.
If you have a good case for the need to make it dynamic as you have above, then let's open up an issue and spec it out...
m
thanks @Stefan Krawczyk I'll give it a try
Open in Slack
PreviousNext
Powered by