Hi, a have built a DAG in hamilton that takes abou...
# hamilton-helpm
Matías Fuentes
04/19/2025, 1:57 AMHi, a have built a DAG in hamilton that takes about 10 minutes to run. It involves typical pandas sync data transformations, as well as a bunch of async LLM API requests. It runs fine from end to end using hamiltons async_driver.
I was wandering if there is a straightforward way to use something like FastAPI StreamingReponses to notify in near-realtime to a client about the progress of the process. For instance, inside one of the nodes I create a bunch of asyncio tasks to process each row of the dataset concurrently with an LLM via API requests, and I would like the client to update a progress bar each time a response arrives. I have multiple nodes that implement this kind of logic, and I would like the client to know in which node the process is currently and the progress of the node.
I am aware that Burr provides streaming support, but I am kinda clueless about how to implement this feature in a simple way.
e
Elijah Ben Izzy
04/19/2025, 4:27 AMHey! So yes, with Hamilton this is doable with lifecycle adapters
Elijah Ben Izzy
04/19/2025, 4:27 AMSee this, for example https://hamilton.dagworks.io/en/latest/reference/lifecycle-hooks/RichProgressBar/
Elijah Ben Izzy
04/19/2025, 4:30 AMBut the q is how are you modeling the request/respinse? 10 mins is a bit long for a request, at that point you’ll likely want a task-based system that:
1. launches a task with an id, returns the id
2. Gives progress updates (either streaming or polling) when hitting an endpoint with the task id
3. Uses lifecycle adapters to write it out to a shared location so it can stream back or return the current state/set of updates
m
Matías Fuentes
04/19/2025, 4:05 PMThanks for you kind reply Elijah, I'm having a blast with hamilton! Lifecycle adapters are exactly what I needed
e
Elijah Ben Izzy
04/19/2025, 8:26 PMGreat! Glad to hear. Yes I think it can help let us konw if you have trouble implementing.