Hi everyone! I'm excited to have stumbled across Hamilton, and have a few questions that have come up as I've been reading the docs: 1. Are there any examples using modal.com as part of the pipeline for remote execution? I was excited to see it mentioned numerous times in the docs, but I didn't see any explicit examples of it in the docs or code samples. 2. Are there naming conventions that are recommended to more explicitly call out the fact that an input is expected to be another function's output? (Sometimes the input is part of the output of another DAG, and sometimes it's just expected to be provided when I invoke things) . As a newbie reading these pipelines, I keep getting thrown by that, and wondering "How do you document what is already going to be computed vs what needs to be provided?" 3. Along those same lines, are there examples of more type-safe declarations of inputs (or outputs) of pipelines? When these pipelines are kicked off, I see that certain inputs are "known" to need to be provided to the executor, but I wish the pipeline was wrapped in such a way as to make it impossible to kick it off without providing those inputs. I will likely create a function or class wrapper to wrap the call to

dr.execute

with a certain set of parameters, but I didn't know if there was a recommended way to do that, or a way that made the pipelines more self-documenting in a type-safe way. From the [async/joke example blog post](https://blog.dagworks.io/p/async-dataflows-in-hamilton):

from hamilton.async_driver import Builder

dr = await (
    Builder()
    .with_modules(async_functions)
    .build()
)

await dr.execute(["joke_response"], inputs={"topic" : "federal reserve"})

There are a lot of strings for me to get wrong there. I'd prefer to see:

joke_response : str = execute_joke_dag(topic="federal reserve").joke_response

Hey! Glad to hear. Good Qs. Will let others chime in, but here’s where I’m at: 1. No examples but it’s not too hard. What do you want to do with modal? There are a few approaches we’ve observed, ranging from calling out to modal within Hamilton, to decorating a hamilton function with Modal, to a more complex integration. FWIW I started at this but didn’t get it out to prod, so there’s room for more powerful work here if you’re interested. 2. Ooh interesting — I really like this idea. If I understand correctly, you’re looking to distinguish between classes of nodes — inputs (from outside), intermediate compute (not queried), and final products (queried by the user). There is no such convention here (it’s BYO), but curious what you find useful! Generally we recommend constructing the DAG and querying the driver using a few inspection methods to determine what’s available — that said, it’s dynamic and not static. Often times the nodes are good in different contexts — E.G. in testing a datapoint should be supplied, but in prod it should be loaded from a db. Some driver functions that will help (the

what_is

functions: https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.what_is_downstream_of, as well as list_available_variables 3. So it doesn’t execute anything if it senses that it can’t (and errors out). That said, we do have some validation methods if you want to bypass calling

execute

, E.G. https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/#hamilton.driver.Driver.validate_execution Regarding type-safety, interesting. There would be a lot of code-generation/magic in what you’ve recommended, although we could probably make it easier to build such a class (E.G. cast it to a pydantic dataframe and use that). Some other options: • You can use function pointers

dr.execute([joke_response])["joke_response"]

• Jupyter magic has some capabilities for this as well: https://github.com/DAGWorks-Inc/hamilton/blob/main/examples/jupyter_notebook_magic/example.ipynb — E.G> making it easier to get the strings right. That said, a wrapper class around the driver that forces it to a dataclass might be really easy to make — there isn’t a good way to get type-hints wired through automatically thought, without manually specifying the types, building a more complex plugin, or using runtime data (E.G. in jupyter)

So you can create your own resultbuilder that could return a pydantic object — we could actually put something like this into the library:

class MyClass(pydantic.BaseModel):
   joke_response: str
    ...

dr = await (
   Builder()
   .with_modules(async_functions)
   .with_adapters(PydanticResultBuilder(MyClass))
   .build()
) 
result: MyClass = await dr.execute(...)
result.joke_response

Wow you guys - this is super helpful, and super fast. I really appreciate the responses! As for what I'm trying to do with Modal: I have some large datasets that happen to be stored in Snowflake at the moment. I need to process some of the rows using a somewhat complicated Python function, but I don't want to upload the python function as a UDF to snowflake and invoke it there. That means I have some steps that look something like: 1. download the dataset and store it locally, perhaps in duckdb. 2. iterate through the dataset in batches of N, and for each batch of N, call a modal function to do the processing. 3. When I get the results back, write the results to another duckdb table 4. Upload the finalized table to snowflake. For the modal stuff, I'm currently creating a generator/iterator that generates batches of rows from my duckdb table, and I am calling for

for batch_result of my_modal_function.map(table_batch_iterator)

and then writing those batch results to a table. This works okay, but A) is a fair amount of hard-coded plumbing for something that I might do somewhat commonly, B) doesn't declare its dependencies well, C) is fragile if there is an exception (this is not resumable/cacheable by default). Hamilton looks like a good lightweight code-first DAG, so I'm pretty excited about that. I'll take a stab at writing the above in Hamilton to see if it helps with any of those issues, but I am hoping to avoid too many newbie mistakes out of the gate, so following a "this is how you do fan-out processing on Modal" example would be helpful. I appreciate you sending me the sample modal code. It does give me an idea as to what a remote executor would look like. I think I'll try to just "use" modal in my steps for now and see how far I get. That notebook stuff looks super handy too! I'll look into that! Thanks @Stefan Krawczyk on the ResultBuilder link. I'm okay to declare types manually for now with a Pydantic object. It just serves as good documentation for myself if nothing else.

Makes sense, thanks for the insight! Yes, agreed that using Modal in your code is probably simplest (for now) — decoupled and gives you control. We did a collaboration with runhouse (basically modal using flexible infra) — you can see an example of a similar approach: https://blog.dagworks.io/p/lean-data-automation-a-principal?utm_source=publication-search

That runhouse example is super helpful! Thank you! In the code, it says this, which gets to another question I have: https://github.com/dongreenberg/gha-data-job/blob/main/batch_process.py

This leverages Hamilton as a few big functions. The piece that runs the embeddings in parallel is all done inside a single function, so those assets are not visible."

Imagine one of my "steps" involves batch processing 100k rows of data of a dataset (in batches of 1000, for a total of 100 calls), and due to a slow external call, takes over 24 hours to process. To get trackability/interruptability/resumability of what is essentially a large map/reduce DAG, should I think about splitting up this single node into N separate dynamic nodes, each of which has their own cache? I am not currently sure how to make such a long-running parallel-batch-task visible to Hamilton and its machinery.

yep! parallelism is what you want here

To add. The way I think about parallelism in hamilton is that “book-ends” some DAG, i.e. allows you to add two functions that then parallelize a DAG over some inputs. So probably overkill if it’s just a single function you want to parallelize. Note: Hamilton has some caveats here with the parallel/collect construct, e.g. the current implementation requires that everything being iterated over/returned can fit in memory --> which means you should be passing pointers around instead… Otherwise you can always parallelize outside of Hamilton.

Ah, that memory caveat is an important one. I am explicitly working around that at present by iterating over chunks of a duckdb database with a generator. So I only need to hold items in memory when I'm shipping them to modal or processing the results (inserting them into my results table). But I guess I could pass references to duckdb table names, parquet files, snowflake tables, etc instead of the dataframe itself... I like the bookends concept. In my case, it feels like each batch is working its way through its own little mini-DAG. Load the batch, process the batch, write the batch back.... So if there was a way to expose that in such a way as to say, "Oh, I got interrupted on step 550, so I'll just do DAG executions for 551-1000 now"...

> So if there was a way to expose that in such a way as to say, “Oh, I got interrupted on step 550, so I’ll just do DAG executions for 551-1000 now”... Yeah this is a little tricky given Hamilton isn’t a full blown task orchestration system. Approaches: 1. have each DAG write to a place upon completion, then in the parallelizable function you check that and skip that chunk if it’s already been processed; consequently in your “collect” function you’d have to adjust for the desired behavior accordingly. 2. hamilton does have a caching mechanism, but it’d need to be extended for this case… 3. you handle the parallelization & checkpointing outside Hamilton

Ah, gotcha - that makes sense. I think I'll focus on making each batch idempotent for now.

Oh and parallelizable and collect don’t currently work yet with the Async Driver (it’s a little weird if you think about it)

Ah, I see! That makes sense. It sounds like that determines where the parallel abstraction happens...

In short: Hamilton is datatype agnostic. E.g. people use Hamilton with PySpark & dask dataframes, which Hamilton helps wrangle the logic, that is then delegated to theses systems to compute over in a distributed manner. Longer: yes, you have full degrees of freedom to determine the data type and what it is. Therefore if you want to get around memory you can use approaches as you suggest.

Excellent.