Hello, Im relatively new to Hamilton, so my questions might be basic, but I couldn't find them in the docs. It would be great if you could guide me in the right direction. The problem I want to solve: My objective is to make a system of models and ETLs that can describe a business. The models need to be defined/developed independently and then connect them together for overall optimization and simulations. I want to chain some models in a way that the output of some will be input of others(with some extra features). with this system I can train models independently and generate simulation and what-if scenarios by running the dag end to end and changing a bit of input or variables. What I considered is having an abstraction for models(since they need to know how to do HP searches for them). Of course, a combination of some models needs to define a model collection. These two objects should be replaceable; the only difference is that the model collection should create the dags based on the model's inputs and outputs. the dags need two have two models(since we train we know the inputs and outputs, while when we predict to create simulations inputs and outputs need to be chained). What I think I need: 1. Can I define dags independently and then merge them later? I saw subdag decorator but was not sure if that's what I need.(for model collection) 2. How can I combine this way of dag definition with classes and abstraction? I need to define interfaces/protocols for the developers to develop some specific methods in their class(e.g. fit, predict). I there anyway for me to create the dags instead of automated version? like I expect in the fit function signature they say what data sources they need, but since this fit model is with, I cant just wrap it around a decorator. 3. How can I have to mode of execution for dag? create two dag in parallel that one has no dependency and other is an actual dag? use config? any other way?

Welcome! So at a high level, this is exactly what Hamilton was designed for. If I understand correctly, you want to be able to: 1. Define pieces of a pipeline 2. String them together 3. Run them under different configurations You want the pieces of the pipeline to adhere to different patterns (interfaces), so you can plug/play. Hamilton does not do class-based inheritance here by design, but it does allow you to do this with modules. Specifically, you would put all the functions in a module and have some sort of interface to that, loosely coupled (ala duck-typing): E.G.

inputs: training_data: pd.DataFrame

outputs: model: Model, metrics: dict[str, Any]

Then you’d effectively write the module so that it respects this interface (+ potentially more parameters). You would then take the driver and chain together the implementations of this interface:

import my_training_module
import my_eval_module

dr = driver.Builder().with_modules(my_training_module, my_eval_module, ...).build()

And the driver checks if you have all the pieces together, by traversing up to the inputs. So you define a DAG in a module that adheres to the interface, and merging is just putting them together in the driver! Often, if you’re building a platform, you’d be writing your own wrapper around driver. This can do simple validation, e.g. something like this:

def create_my_driver(training_module, eval_module, ...):
    validate_training_module(training_module) # creates a temp driver, checks inputs
    validate_eval_module(eval_module) # ditto 
    return driver.Builder().with_modules(training_module, eval_module, ...)

We found that this is much simpler to reason about than classes, which have an added layer of state that people almost always use. Hamilton + functions imply statelessness, which makes it simpler to manage/track/repro. I think this gest at (1) and (2) above. I”m not 100% sure what you mean with (3) — mind drawing out what you’re trying to do?

Thanks a lot for the reply. Let me provide the basic idea. Here is the link. Lyft has a really special way of solving this problem(which Im not interested in) but I want to solve a similar problem and think Hamilton is the perfect match. since I want to make a library / framework which needs to be model-independent, I need to depend on an interface, and either ask users to tell me what is the input/outputs of each model or make a decorator that parses that and use hamilton to create the node for me. I feel I need more control over how the dags has been created, but don't know where can I find that info. User will create a model lets say `marketing_ctr`that has

fit

predict

and some other methods(based on the interface( I need the interface because the users can be responsible on their model validation and hp search without knowing about the rest of the system) and in fit and predict function they say what inputs they need to create the output. for point number 3 since this idea has two phases of training and inference and in the prediction phase all the models can be trained in parallel, and in inference the dag dependencies will be the most important factor. that's why I was asking do I need two dags(one for train and one for inference) or if should I use

when

conditon.

Ok, think I get what you’re trying to do. For the two-dag issue you have two choices: 1. Use

config.when

— useful if there are a few pieces that vary between them 2. Use separate modules (E.G.

train

and

inference

) — useful when there are more differences between them. This post goes over the options — think it will be pretty relevant to you: https://blog.dagworks.io/p/feature-engineering-with-hamilton. Just to be clear, if you had a specific decorator for this, what would it look like? Subdag, etc… can all help, but writing it down might make it clearer. I think that implementing a module with a certain shape will likely be enough, but you could probably create a custom decorator on top of other ones for your specific case, if the interface is constant/limited. Note this also could take in an object if you really want to — there are some options here. But yeah if you write out your ideal code I can help you figure out how to structure it with Hamilton (and how much you need to dig into how the DAG is constructed)!

without knowing about this library, I made a super simple similar thing based on networkx. There, the interface was something like this

class Base:
    def __init__(
        self,
        *,
        parents: list[str],
        children: list[str],
        obj: Any,
        obj_type: Literal["model", "func"],
        name: str = None,
    ):
        """Base class

        Args:
            parents (list[str]): 
            children (list[str]): 
            model (Any): Any model
            name (str, optional): Name. Defaults to None.
        """

        
    @abstractmethod
    def fit(self, ) -> None:
        "The input data should match the `parents` schema"
    
    @abstractmethod
    def predict(self, ) -> dict[str, DataFrame]:
        "This function should match the `children` schema"
    @abstractmethod
    de hp_search(self, ):
       ...

    def train(self, data: DataFrameDict):
        self.fit(data)
    
    
    def __call__(self, data: DataFrameDict) -> DataFrameDict:
        return self.predict(**{name: data[name] for name in self.parents})

and based on this class I created 3 types which is transformation functions like ETLs, models and model collections. users based on their tasks inherit one of these 3 classes and implement their logic like this:

class CTR(Model):
    def __init__(self, model: Any):
        super().__init__(parents=['user', 'price', 'competition',], children=['purchase'], model=model, name='ctr')

    def fit(self, data):
        X = pd.concat([data['user'], data['price'], data['competition']], axis=1)
        self.model.fit(X, now_dict['purchase'].purchase)
    
    def predict(self, user, price, competition):
        X = pd.concat([user, price, competition], axis=1)
        return {'purchase': pd.DataFrame(self.model.predict_proba(X)[:, 1], columns=['purchase'], index=X.index)}

ctr = CTR(model=ctr_model)

or for model collection something like this

model = ModelCollection(
    models=[ctr, purchase_filter, freq, sev, p_and_l],
    name = 'model'
)

which is has the same idea as the driver(all of these are defined models, or ETLs or other model collections) and it creates dag like the attached image. you already can see some of the problems with this approach, like in fit it always uses the whole data, it depends on dataframe, and it executes the code in the same place as it creates the dag...which Hamilton has a better solution for them. I think my library should sit between user defining their models and hamilton creating the dags to make it. after finding this library I thought I can recreate this solution with a better backend, but could not solve the problems with classes or having two modes of execution

This is great — afk for a bit but I’ll take a looks nd see if I can sketch out some pseudocode!

Thanks alot. I think I already have a better idea of what is possible. Do you have any examples of the Hamilton module that has been implemented in a class? It's ok if it's from the internals of the package. Just need to see how complicated it is.

You got it! So we don’t — all the examples are stateless functions. You can do pretty much anything you want with the internal API, but there’s no guarentees that it’ll be stable. If you’re referring to (2) above, I’m thinking something like: 1. Define the class you want people to implement 2. Define a module that everyone will use based on the class — E.G. have it pass an instance of that class back and forth and call it That said, it’s problematic because classes bundle together state + interfaces (E.G. inheritance requires you to also deal with object state) — so you have none of the nice guarantees you’d get from a normal set of python functions.