Hi All. I am new to Hamilton. My use case is not machine learning but rather converting formulas from spreadsheets to Python code. Hamilton is an option because a formula in one cell depends on the output of a formula in another cell and so on. So there is a DAG effect. It seems I will require to use one function per rule and pass in Series as opposed to dataframes. But then I will have to reassemble the dataframe at the end somehow... Is it okay to just pass the whole dataframe to the function which then appends a column with its transformation and the next function does the same etc. At the end I will then have my dataframe with Transformations?

hey @Shoaib ! Yes I've thought about that case -- especially since you can now write python code in a spreadsheet ...

It seems I will require to use one function per rule and pass in Series as opposed to dataframes.

So with Hamilton you can choose at what level of granularity you want to think at - so you can mix and match series or dataframes.

Is it okay to just pass the whole dataframe to the function which then appends a column with its transformation and the next function does the same etc.

One caveat is that you generally don't want to mutate the input because that can lead to bugs.

But then I will have to reassemble the dataframe at the end somehow...

There are many ways to do this. You can make a function to take in the columns and output a dataframe your self. If it's the last thing you can implement your own custom result builder or the standard pandas one.

Thanks @Stefan Krawczyk . I'll keep all this in mind. I'll experiment and go through the documentation in further detail too.

@Shoaib the modules on https://www.tryhamilton.dev/ would be a good start

In my current project, we did exactly that, we load, combine and transform data coming from xls+csv. I made each transformation into a function and used the graph to communicate with the user how the transformation was done. The user was able to understand the flow of transformation and made me change some steps. I used the extract_field to expose an existing function that returned multiple dataframe so it's clear in the graph where some data comes from.

The dag works quite well for my use case. I was just wondering is there a way to output all the functions that the driver executed. Or if not after execution, then just the list of functions in order that will be executed based on the parameters to the driver. There is dr.list_available_variables but I am not sure if it guarantees to give u the list in dependency order.

If you write a simple adapter, you could capture the execution order*. See lifecycle adapters in the docs or the dagworks blog. It shouldn't be that many lines of code.