Hey all, I would like to know if its possible to create loops with hamilton like this image down. Essentially, I want to use different functions for each step on my pipeline (read data, preprocess, train/test split, etc). I want to be able to execute the train/test/split multiple times, saving the results of each model`s inference, and then displaying/saving one of the models (with a fixed seed). The reason I want to execute train/test split multiple times is that I want to test different seeds, and then calculate mean/std values for N models. At the same time, I would like to execute this pipeline in a single step (with N=1, for instance), not activating that loop.

Hi @João Paulo Vieira! I've used Hamilton for similar ML pipelines and I'll share solutions that worked for me. But before that, a few assumptions: 1. Hamilton uses a directed acyclic graph (DAG). Loops/cycles from a node output to the input of an upstream node (e.g.

inference -> train/test split

) 2. as I understand, you don't need to pass values from

inference

to

train/test split

, you just want to repeat the operations 3. it seems that the steps

train/test split -> model -> inference

are independent and parallelizable. 4. Do you know how many training loop you want to execute? # Solutions I advocate for adopting the simplest solution that works and gradually adopting Hamilton features. ## Solution 1 Multiple

dr.execute()

calls and

for

loop

# build Driver
dr = Builder().with_modules().build()

# prepare dataset; has shape {"preprocess": VALUE}
prepared_dataset = dr.execute(["preprocess"], inputs=...)

# loop over train/test split branches
for _ in range(10):
    iteration_results = dr.execute(
        ["inference"],
        overrides=prepared_dataset  # has shape {"preprocess": VALUE}
        # inputs = {...} you could pass iteration specific inputs
    )
    # do something with iteration results

Then you can save results on each loop iterations (or you could have a node that does this). I suggest looking into materializers and the ## Solution 2 You can use Parallelizable/Collect to run everything in a single DAG. This would require modifying the dataflow definition. In my experience, this is not desirable for ML because parallelization introduces bottlenecks (e.g., you can't train 5 XGBoost models in parallel on a single machine). You're better off leveraging the built-in parallelization in polars/numpy/XGBoost/PyTorch to train one model at a time, but efficiently (in other words, use Solution 1) --- Finally, caching is probably something that will be valuable for your project

Thank you so much @Thierry Jean, your answer has helped me alot. I'll go with the first solution, since it makes sense to me.