Hi, is there a way to read back the parameter values from a parameterized node? For example

@parameterize(
    SOME_OUTPUT_NAME=dict(n_off_date=source('SOME_INPUT_NAME'), shift_by=value(1)),
)
def date_shifter(n_off_date: pd.Series, shift_by: int = 1) -> pd.Series:
    """{one_off_date} shifted by shift_by to create {output_name}"""
    return n_off_date.shift(shift_by)

...

node = driver.graph.nodes['SOME_OUTPUT_NAME']

Is there a way to get

shift_by

from

node

?

Good q! So, currently it’s not attached explicitly — you might be able to dig around for it, but I think these are two cleaner ways: 1. Encode it in the name you provide (e.g. join with underscores) —

SOME_OUTPUT_NAME_<i>

2. just remember a mapping — us e as it of name -> shift values That said, I think this could be useful, and we could encode the parameters in tags (which are accessible). Mind opening up an issue?

🤔 shouldn’t it also be recoverable from the callable function?

Better to make it explicit — the fact that it’s a partial is an implementation detail that we don’t want as part of the contract

I don’t think that’s the right approach.

Anyway mind creating an issue @Vadim Ogranovich and @Stefan Krawczyk and I can figure out the right implementation there?

@Vadim Ogranovich what would be helpful is a sketch of how you want to use this information, that way we can think through what’s a good API to expose this without being overly dependent on the internals, please 🙂 (though I’m not opposed to this being a power feature that comes with caveats if you reach into the originating functions…)

I was actually hoping the bound vars would be listed in the

default_optional_values

attribute of the node. It would make it consistent with the use of the attribute in the regular, aka function, node. There should be no difference between

def a(x = {}): ...

and

@inject(dict(x=value({})))
def a(x): ...

@Vadim Ogranovich this is helpful. Though in the above example, you want to get the bound value not from the default value on the arg, but from the decorator, right?

yes, if you are referring to the second (decorated) example.

In the first one though. There's the default value, and the then value in the decorator.

the upshot is to be able to manipulate

x

via the node object. For vanilla nodes one can already do it via

default_optional_values

. For parameterized nodes each generated node should have its own

x

and I want to be able to get hold of and manipulate it. Does this make sense?

What do you mean by manipulate? I think surfacing is possible yes. But in this case should it be classed under optional? Or a new construct of bound values?

I am not sure what surfacing is. To manipulate means to alter its state. For example I might want to add an item to the dict, e.g.

x['foo'] = 1

, outside of the dag execution.

So yeah, by “surface” — ust how to expose it so you can leverage it. That said, I’m not sure that those two are the same (insomuch as Hamilton thinks about it) — there’s a subtle difference into the DAG they turn into. The first one

def a(x: dict={})

is effectively saying “if

x

does not exist (either as an upstream node or an input), then it has a value of empty dict)` The second one says: “the

a

node has a fixed value of

x

” Thus if you provide

x

as an input it’ll work for the first one — but for the second one I don’t think we can use it as an input. To be clear though, by “manipulate” — do you mean change the parameter at runtime? E.g. dynamically alter the inputs?

And sorry this is the "first one" I was referring to.

oh, I see. Yes, the parameter value should come from the

value(...)

of

@parameterize

. My apologies, I started to use

@inject

as the baseline example. With an implicit implication that it should apply to

@parameterize

too.

@Vadim Ogranovich have you seen https://hamilton.dagworks.io/en/latest/reference/decorators/resolve/ ?

Re "The first one

def a(x: dict={})

is effectively saying “if

x

does not exist (either as an upstream node or an input), then it has a value of empty dict)` The second one says: “the

a

node has a fixed value of

x

” I understand this distinction, and it's an issue indeed, but it doesn't affect my use case. You guys can keep the two semantics and each of them will work for me.

yep all good. I’m just trying to mention current design decisions — which we could always think about adjusting.

I was aware of

@resolve

. It "might" do the trick, but it's too bulky for my use case. I'd rather do without

@parameterize

and clone nodes manually.

I’d rather do without

@parameterize

and clone nodes manually.

Side comment: I have thought about “compiling” away some decorators, and outputting verbose code — but there really hasn’t been a use case for it.

To re-iterate

default_optional_values

on vanilla nodes do the trick for me. I am just "asking" for more comfort 🙂

Sounds good. Let’s get an issue up with the expected UX you’d want.

Yes, compiling away is an option I keep in mind should

@parameterize

not going to work out.