Hey folks, hope you're all well. I'm interested in the using the new Async version of Hamilton for a microservice that would primarily pull data from an API in response to user input via a dashboard, process it and return the results to the dashboard. Complexities include that the microservice must also operate independently as an API and the data processing library be reusable as a dependency in other projects and as a CLI. I have in mind putting async Hamilton inside a FastAPI such that the inputs might be little more than URL components and the dataflow pulls the data in some node. That seems appealing because in circumstances where the data needs to be supplied from another source, it could (I am thinking) be passed in as overrides instead but still validated using the new validation decorators. It's been a while since I wrote with Hamilton in anger. Would be in pointers to any recent published examples that might fit that pattern and or feedback/suggestions to the idea. Many thanks!

Hey just some quick thoughts before I jump to bed — will respond in 8+ hours… 1. I assume you’re using async because of some I/O libraries? Note: with async Hamilton only functions that are async need to be async — the rest can be standard python. 2. yes you could run the same DAG in fastAPI and behind a CLI. If there would be differences you could use familiar constructs to swap/change them. > That seems appealing because in circumstances where the data needs to be supplied from another source, it could (I am thinking) be passed in as overrides instead but still validated using the new validation decorators. Materializers is one way, overrides is another, or simple @config.when, or separating out modules — e.g. in fastAPI you have dataloading module + core module, in CLI you only have core module and you pass things in as inputs… > Are materializers

Parallelizable

? you can annotate functions in parallelizable block with @load_from / @save_to. But you can’t “override/inject” into a parallelizable block (we wouldn’t know which one to overwrite). > Any thoughts on whether it would be better to load data via concurrent tasks in a materializer function or to load via

Parallelizable

? What’s your use case here? Depends. Do you want extra visibility? Or a simpler DAG? or? Note: Parallelizable doesn’t have an async executor IIRC.

Materializers is one way, overrides is another, or simple @config.when, or separating out modules — e.g. in fastAPI you have dataloading module + core module, in CLI you only have core module and you pass things in as inputs…

Yes of course, maybe modules are neater with a driver factory pattern. I've done something like that before.

you can annotate functions in parallelizable block with @load_from / @save_to. But you can’t “override/inject” into a parallelizable block (we wouldn’t know which one to overwrite).

Fair enough. I think I would override/inject downstream of the

Parallelizable

loading if I did it that way.

only functions that are async need to be async

I don't quite follow what …?

What’s your use case here? Depends. Do you want extra visibility? Or a simpler DAG? or?

Note: Parallelizable doesn’t have an async executor IIRC

If no async executor then sounds like putting an existing async paginated loader in an async node is maybe the way to go. Primary use case is packing some fiddly, custom data processing logic behind a FastAPI. I did something similar about six months ago with the async driver and a transformer model and am keen to try and improve on it with the updated version. Visibility is not critical, but since you have your new UI, I would like to try integrating it if circumstances permit. It is important that secondary use cases (CLI/depndency) can be handled from the same codebase, most of which is already async.

I don’t quite follow what …?

Yep sorry. Just saying that you don’t need to write all your functions with

async def

. Which it sounds like you already know. 🙂