Hi everyone, I'm starting to use Hamilton in my project and have a few questions. 1. How are you integrating Hamilton with Hydra for configuration management? I couldn't find any examples in the documentation. 2. My project involves both training and inference processes that run in batches. Is there a way to create a loop of nodes in the DAG so I can reuse the same nodes for each batch within a single execution? Currently, I have a node that iterates over all batches and executes the necessary functions, but this approach makes visualization and debugging more challenging. I'm still going through the docs, so apologies if these questions are already covered there. Thanks for the great project!

@Thierry Jean might have thoughts on hydra here.

My project involves both training and inference processes that run in batches. Is there a way to create a loop of nodes in the DAG so I can reuse the same nodes for each batch within a single execution? Currently, I have a node that iterates over all batches and executes the necessary functions, but this approach makes visualization and debugging more challenging.

Do you mind sharing a sketch of what you’re trying to achieve and how you’re trying to use Hamilton please? That would help me respond. In case it’s helpful here’s one (notebook) that could be relevant here.

> 1. How are you integrating Hamilton with Hydra for configuration management? I couldn't find any examples in the documentation. At a high-level, those would be the component of a Hamilton project with Hydra: • Dataflow definitions (Hamilton): those are

.py

files containing Python functions to define your Hamilton DAG • Driver code (Hamilton): in Hamilton, a

Driver

is the object that takes the dataflow definition, creates a graph and execute it. It's similar to the "pipeline-level" code. • Configs (Hydra): config files are helpful to build the

Driver

and set values to execute it. Typically, the config can contain: environment variables, secrets,

Driver

config, materializer definitions, execution inputs & overrides, etc. TL;DR: Hamilton defines the data transformations and pipeline, Hydra configures the pipeline execution As an example, I used Hydra and Hamilton in this project: https://github.com/zilto/ordinal-forecasting-digital-phenotyping/blob/master/scripts/train_tabular.py There's a recent blog release about Metaflow and Hydra that could be relevant: https://netflixtechblog.com/introducing-configurable-metaflow-d2fb8e9ba1c6 Things said about Metaflow are relevant forHamilton.