https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hi, I have a project I'm currently working on at m...
# hamilton-help
j
Hi, I have a project I'm currently working on at my company and I would like to take this opportunity to try Hamilton, and introduce it to the company. I am new to the library but I've been following your work for a while and really love what you are doing. Read a lot of the docs and blogposts (mostly @Thierry Jean's ones šŸ˜‰ ) and I'm super excited to start using it! I was wondering if you could provide some guidance to my use case, and even challenge whether or not it's a good use case for Hamilton in the first place. The use case My company has a machine learning mono repo where a lot of cloud functions are being duplicated and repeated instead of having 1 source of truth for them. I want to build a package with all of these "util" functions using Hamilton that would get used inside our various ML projects. (load/saving to buckets, to our data ware house, model deployments, etc). Since we will also be using AWS in the future, I was thinking of creating 2 layer API approach where I provide high level functions: ex 
upload_blob
, 
download_blob
, and hide the implementation of either GCP or AWS. Maybe 
gcp
or 
aws
could be passed as an input to those functions, and I'd build the logic with Hamilton to load the proper module. Same for our data warehouse and deployment functions. In the screenshot below, you can see an example of some gcloud storage functions I quickly put together. Questions 1. If I am just building functions for just GCP, what might be some advantages of using Hamilton vs coding this in "plain Python" and just using the 
gcloud
library? Seems like the latter would be simpler. a. I understand the observability features I'd get with Hamilton, but saving to GCS isn't a complex data transformation like you could find with feature engineering for ex. b. Would it make sense to use Hamilton at all for this use case? It would make sense to me if I were implementing end to end data transformations with the last step being "save to GCS". But in my use case, I'm only implementing "saving/loading to GCS" without the middle transformations. Whatever transformations come before or after wouldn't be using Hamilton (...yet) since other people in the team would be writing them and wouldn't be onboarded with Hamilton yet. Maybe Hamilton would be "overkill" for this? 2. Assuming Hamilton is a great choice for my use case, and that I'm moving forward with a 2 layer api approach, I'm wondering what features you would suggest for implementation? a. Should I opt for a basic module -> driver -> execute approach for each function I may want to provide to start? Would I be missing out on some of the more advanced features like materilizers, DataLoaders/DataSavers, config, etc? Any pattern you would recommend for me? b. Do you have examples of anything resembling my use case? For this 2 layer API approach, I was looking at this data_loaders example where 3 different implementations are done and the driver then calls either one of the modules to get executed. I have lots of other projects going on that might be a better fit for a good demo perhaps: ex - Airflow DAG where I am running batch model predictions that are saved to a table. Let me know if I should save my Hamilton efforts for that, or if I should start using it now. I appreciate your help and let me know if you need more information Thanks for all your work!
šŸ‘‹ 1
s
1. If I am just building functions for just GCP, what might be some advantages of using Hamilton vs coding this in ā€œplain Pythonā€ and just using the 
gcloud
library? Seems like the latter would be simpler.
a. I understand the observability features Iā€™d get with Hamilton, but saving to GCS isnā€™t a complex data transformation like you could find with feature engineering for ex.
b. Would it make sense to use Hamilton at all for this use case? It would make sense to me if I were implementing end to end data transformations with the last step being ā€œsave to GCSā€. But in my use case, Iā€™m only implementing ā€œsaving/loading to GCSā€ without the middle transformations. Whatever transformations come before or after wouldnā€™t be using Hamilton (...yet) since other people in the team would be writing them and wouldnā€™t be onboarded with Hamilton yet. Maybe Hamilton would be ā€œoverkillā€ for this?
You can use Hamilton here, but yes it might be overkill. The upside of using Hamilton is that if you wanted to add logging, get that nice picture, etc. you can use the framework, or if you wanted to feature flag different implementations you can do so.
1. Assuming Hamilton is a great choice for my use case, and that Iā€™m moving forward with a 2 layer api approach, Iā€™m wondering what features you would suggest for implementation?
a. Should I opt for a basic module -> driver -> execute approach for each function I may want to provide to start? Would I be missing out on some of the more advanced features like materilizers, DataLoaders/DataSavers, config, etc? Any pattern you would recommend for me?
b. Do you have examples of anything resembling my use case? For this 2 layer API approach, I was looking at this data_loaders example where 3 different implementations are done and the driver then calls either one of the modules to get executed.
The two-layer approach should allow you to change things later without much hassle so it lowers your risk here. Materializers are basically just a way to centralize some logic and/or inject nodes into the hamilton DAG. So start without them, and once you see repeated code, etc. then reach for themā€¦ IIUC the two layer API with hamilton would be:
Copy code
# higher level API
def upload_gcs_blob(something) -> dict:
   # lower level API
   dr = driver.Builder().with_config({...}).with_modules(gcs_hamilton).build()
   result = dr.execute(["upload_blob"], inputs={...})
   # massage result to conform to API
   return result
I have lots of other projects going on that might be a better fit for a good demo perhaps: ex - Airflow DAG where I am running batch model predictions that are saved to a table. Let me know if I should save my Hamilton efforts for that, or if I should start using it now.
No strong answer here. But definitely a fit for airflow (@Thierry Jean has a blog on that).
Hamilton DAGs are composable from modules, as seen in that data loading example, so if data loading and saving is something that could then be used in a Hamilton DAG later on then maybe itā€™s useful to start with Hamilton now?
j
Thanks Stefan for answering so promptly. This clarifies my understanding The code you wrote for 2 layer api is indeed what I had in mind. I'll sleep on it for a final decision. I'm thinking: since the purpose of the Hamilton intro I am doing is specifically to drive adoption towards the tool, showcase Hamilton value, and convince my managers to onboard with it, it might make more sense to start with a full dataflow. I also want to demo this to the data engineering team - so showcasing an full Ariflow DAG makes a lot of sense Thanks again for making this clearer for me
ā¤ļø 1
šŸ‘ 1
Open in Slack
PreviousNext
Powered by