Has anyone established a good workflow to work with an AI editor like Cursor/Windsurf and Hamilton in Jupyter Notebooks? I'm struggling with the lack of native support for notebooks in these editors + having some difficulties "teaching" these AI how to use Hamilton properly @Thierry Jean

Can you provide them with some “markdown context”? It might be that we need to create some file that helps them understand the paradigm with some examples.

Yes. That would help. If I'll find something that works as one-shot I'll post it here. In addition, most AI IDEs don't support AI in their jupyter notebooks, which makes the whole development with hamilton magic a pain 😔

Wasn't there someone who asked for a pdf version of the docs for an air-gapped system? Maybe a recent build of that could help LLMs... @Stefan Krawczyk

@Iliya R I suspect the token count might be a bit large 😆

We have a claude subscription from work, and it allows uploading files to "project context", which supports fairly big pdfs (from experience). I can give it a try with the hamilton docs and see what happens

Notebooks are poorly supported because

.ipynb

are just JSON files. Tokenizing JSON is super expensive with

" [ ] { }

each taking a token. Sadly, I don't have any silver bullet here

The problem with that is that it doesn't know when to use parameterize (or anything else) given a description that doesn't mention that explicitly

A challenge with LLMs and AI-enhanced IDEs, is determining what kind of prompts should work and how to collaborate with the LLM efficiently. For context, what type of prompts did you try and what were the results (the good and the bad)? # Generating the DAG Queries like "create a DAG to load web pages, chunk them, and embedded them" are very challenging. With good prompts and examples, we could work towards "generate function signatures for a valid Hamilton DAG". # Filling the node functions IMO, writing the skeleton of the DAG is a "human-first" job. I had good success writing function signatures + docstrings and then using LLMs to generate the body of the functions. (this could be a 2nd step after a successful 'Generating the DAG' operation) # Function modifiers, materializers, etc. Once you have the DAG and the filled body, LLMs with relevant context should be able to answer "parameterize the function

monthly_sales()

to be applied per team and per department". This follows the idea that it's mostly human's role to specify the DAG shape On the Hamilton side, our best effort is to create an LLM-friendly version of the docs + improve the docstrings of public interfaces

Hi @Thierry Jean Are there any updates on this?

Sadly, no 😕

You might want to consider a plain text notebook format like Quarto. That's mostly just markdown, which LLMs like. Note though that the state of your document is not embedded in the document itself like with ipynb, so you'll likely also want caching enabled in Hamilton.

@Thierry Jean What about a Hamilton mcp server? Here is one for polars. Should be easy to adapt to hamilton https://github.com/HotTechStack/polars-docs-mcp/tree/main

@Volker Lorrmann ooh, that's a cool repo! I had a similar idea of using Python runtime to inspect objects. Will try to implement something similar