🎄 Happy Holidays 🎆 🕎 🕯️ Everyone! We’re excited and thankful for you all being part of this community 🙂 Speaking of community, I’d love a few volunteers to try a release candidate that includes telemetry capture (https://github.com/stitchfix/hamilton/issues/248). I’m really excited for it, since it’ll us understand what parts of Hamilton are being used so we can know where to invest effort! To test it:

pip install sf-hamilton==1.13.0rc0

. I have tested it extensively myself, but I would love to confirm that: • this does not impact anyone’s Hamilton code. • opting out is straightforward (see this section that will be added to the README). Feel free to DM me, leave comments in https://github.com/stitchfix/hamilton/pull/255, or in a 🧵 on this post 🙂 Thanks in advance!

Hi All. We’ve added two new examples to the repository — looking for feedback here: 1. a basic example showing Hamilton applied to a time-series forecasting Kaggle task. 2. examples showing how one might extract data to get it into Hamilton for downstream use. Are these examples useful? What else could we add that would have been helpful for you?

Happy Friday. New release candidate is up:

pip install sf-hamilton==1.15.0rc0

It has adds the ability to directly see the DAG in a notebook: and includes a minor fix for optional arguments. I will push this Sunday/Monday assuming no issues in testing (don’t anticipate any).

Hi everyone! We’re working on revamping the Hamilton documentation and moving to

readthedocs

(see migration progress here - thanks to @Sarah Haskinsfor the contribution!). We’d love any feedback/thoughts you might have about what could be improved. Feel free to respond in this thread or create a github issue with your thoughts. Thanks for your help!

Question, why do you need dbt fal to integrate with dbt? Also, does this make sense for a pandas etl stack? Seems like they expect you to run dbt then fal when executing dbt fal models. In my case heavy lifting is done in python and additional business logic in mvs in SQL, so I don't get how having to run dbt then fal would work. I'd like to use Hamilton to keep the pandas organized and dbt for the SQL.

Fwiw we are heavily optimizing for performance on postrges

Anyone interested in providing some feedback on a feature engineering example we’re working on that shows how to share feature definitions in an offline and online setting? If so, I’d love some feedback on some documentation (👍 , 😐 , 👎 ) that I’m writing that shows how to use Hamilton to do that. The example can be found here on a this branch; I’m struggling to hit the right level of detail without being overwhelming/boring, but with enough to help someone out… thoughts and comments appreciated - thanks in advance! (PR for it can be found here that you can leave comments on too).

hi folks, I'm considering migrating a training pipeline to hamilton instead of a custom pipeline. one feature that it has at the moment is that most steps are cached to disk, so that you can skip some leading step later on, or continue after failure, and stuff like that. I was wondering if there's any particular approach for this that y'all would recommend to implementing it with Hamilton. so far my thinking is that I could write whenever I need to inside the node, and then have conditional

@config.when

type nodes for when it's supposed to be computed or loaded. note that this is different from "only compute if not cached" type of scenario (which I may also want at a later point, but now I'm interested in being able to skip some nodes and simply fail if cache is not there). any thoughts would be appreciated!

Hello, I am looking for a way to parameterize hamilton functions from separate config files or simple python modules, such that I can have different DAGs based on different configs. This way, I can have a common library of hamilton transformation functions and reuse them to get multiple versions of DAGs based on usecases. Any suggestions on how to achieve this?

Does anyone here use VSCode? If so, @Thierry Jean is working on an extension for Hamilton. It’s still in alpha and he’s actively developing it, but if you’re interested in adding your thoughts/feature requests - drop them in this discussion here.

Apologies if it's been asked before. Is there a decorator we can use to exclude a function in a module from being parsed by Hamilton? I have a helper func with one argument, Hamilton tries to resolve it. To prevent the error I create a dummy function. The use-case is: I wish to utilise the helper function without declaring it in the graph, however I want it to live close to the main function in the same module.

My pipeline requires running the same query over two date ranges and using the resultant datasets separately at various points in the graph. In other words, I would like two artifacts for two distinct arguments to the same function. The two date ranges are parameters to the DAG. Is there a pattern for doing this? Many thanks 🙂

watching @Elijah Ben Izzy at PyCon Lithuania about Hamilton ⭐

Hey everyone - I wrote a new post about Hamilton’s lineage capabilities — https://towardsdatascience.com/lineage-hamilton-in-10-minutes-c2b8a944e2e6

Interested in making custom

DataSaver

subclasses. Saw the docstring note that it is currently not a public facing API and may change. Would you happen to have some visibility into when you plan to make this a custom facing API?

Happy Friday! ... I was just curious to know what IDE/tooling is the hamilton team using to work on Hamilton? i see a little snippet relating to pycharm in the developer setup guide

Hey folks! Wanted to share some code that I used for to crawl a bucket in s3 and download all the files locally for analysis. This uses the new `Collect[]`/`Parallelizable[]` API, and is idempotent — it’ll start where it left off! Note this can be done quickly with the awscli, but in python it gets more messy, and running in parallel gets uglier. That said, the value of this is: 1. Visualizing/tracking the DAG 2. parallelism handed to the framework 3. Easy insertion of transforms/other changes - I have a transformation component that I added in (not shown in the code) to save the files in a different format 4. Workflow — this has an easy caching mechanism, but there are lots of other ways to pull something more sophisticated off. Code/DAG in 🧵

This message was deleted.

set the channel description: This is the one channel that will always include everyone. It’s a great spot for general Hamilton conversations. We’ll redirect you to a channel if there’s an appropriate one. For promotional content please use #be-shameless.

In case it’s helpful we’ve been building some

hamilton tutorials

in an aptly named repository called https://github.com/DAGWorks-Inc/hamilton-tutorials. The idea is to help give some ideas/ways to structure/iterate on Hamilton code to accomplish something. They are

notebook

focused because that’s an easy way to wrap documentation and code. For example, you can run them in google collab easily. https://github.com/DAGWorks-Inc/hamilton-tutorials/tree/main/2023-10-09 is the latest version. It contains three directories: • one for PDF summarization - i.e. LLM workflow to summarize a PDF. ◦ contains a single notebook that walks through some ways to build a LLM workflow. • one for the Titanic Data set - i.e. a data/feature engineering + ML pipeline workflow ◦ contains two notebooks ▪︎ one walks through data & feature engineering. ▪︎ the other fitting a statistical model and builds off the first. (FYI the model isn’t sophisticated). • one for some Hamilton reference code snippets ◦ this should mirror what’s in the docs, but it could be useful for you to save time looking something up. Pull requests/comments/issues suggestions welcome.

Hi all. I wanted to surface a few best practices that we have in the documentation that would be helpful to browse if you’re getting started: • https://hamilton.dagworks.io/en/latest/concepts/best-practices/ It covers things like: • migrating to Hamilton • function naming • output immutability • code organization If there’s things missing, or topics you’d like it to cover, please create an issue, or comment here 🙂

hey all! So we’re excited for all the requests to do more with visualization. We’re trying to take input for the roadmap here and want to know if anyone else that has thoughts/needs/requirements for extending the visualization capabilities that Hamilton currently has. So let us know if that’s you, to (a) include you on discussions and (b) get requests for feedback from you.

BTW does anyone want to use Hamilton to orchestrate command line tasks intermingled with regular python code? We’ve got a discussion related to SLURM here — with a generic run a command line prototype at the bottom. Would love any thoughts/input.

Hey everyone! Recently: 1. I got tired of setting up experiment tracking boilerplate code for MLFlow or Weights&Biases 2. I was excited to try the recent FastUI framework to create web apps UI on top of FastAPI (i.e., competing with Streamlit, Shiny, Dash, etc.) So I decided to build a lightweight local experiment manager plugin for Hamilton! You can access it via

hamilton.plugins.h_experiments

see the example here I go into more details about the building process in our latest blog: https://blog.dagworks.io/p/building-a-lightweight-experiment The best part about open source tools like FastAPI, FastUI, and Hamilton is that you own your platform. It's easy to get started and you can incrementally add the features YOU need!

showcase.gif

FYI - meetup thread can be found here - https://hamilton-opensource.slack.com/archives/C03M328JBTM/p1708449808237289

Office hours here - https://hamilton-opensource.slack.com/archives/C03AJNGDGQL/p1709659885737809

FYI - https://hamilton-opensource.slack.com/archives/C03M328JBTM/p1710888451833979

Have to say, although I missed the hangout @Roel Bertens’s your template and work for the demo has been a huge help to a newbie like me trying to get a feature generation pipeline migrated over to Hamilton. Thank you very much!

I was playing around with OpenAI’s GPT functionality. So I created a quick one to help translate code to Hamilton — it still hallucinates a bit, but could be something to try — https://chat.openai.com/g/g-Yl6yWH39S-hamilton-code-engineer would love feedback. If it’s wrong you tell it and it usually corrects itself, though I still need to teach it about all the decorators…