This message was deleted.

All good, we love answering questions! I'll let you in on a secret -- you can pass any valid python object to the driver. E.G. if you define your node as:

def with_inputs(input_df: pd.DataFrame, value_a: int, value_b: str) -> pd.Series:
    ...

Then you can run it like this:

driver.execute(
    ['with_inputs'], 
    inputs={
        'value_a' : 1, 
        'value_b' : 'foo',
        'input_df' : pd.DataFrame(...)})

If you want it to be a dict of values rather than a few specific ones, you could also pass in/declare a dict (but IMO specific values are more optimal here).

awesome. that seems like it restricts me now to having to treat the input dataframe as a dataframe and not as a set of columns? i.e. will I be able to later refer to columns of input_df as nodes in some other function? I guess I’m struggling to see how the DAG works to go from

input_df

to say some

def output_column(input_column_inside_input_df: pd.Series)

or would I have to do something like

extract_columns

to expose the columns in

input_df

to the DAG?

@Zouhair Mahboubi

<http://df.to|df.to>_dict('series')

I think will do the trick for you. — it will return a dict of { col_name -> series} that you can then pass in as input.

Yeah — to be clear hamilton will think of it as a df if you pass it in as a df, not as a bunch of series. The to_dict method will allow you to input it as a dict of series instead, or you could use extract_columns within the DAG. I think @Stefan Krawczyk ‘s approach is cleaner/more explicit.

I guess that works as long as the DF is not too huge. Should be fine in my usecase. Thanks !

It shouldn’t increase memory — I assume it should just be references.