https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
This message was deleted.
# hamilton-help
s
This message was deleted.
👀 1
s
@Arthur Andres 
put_features_together
could be replaced by - https://hamilton.dagworks.io/en/latest/reference/api-extensions/custom-result-builders/ if it’s the last step.
If it’s not the last step it’s a little more complicated (open to better APIs here) and I think pairing inject with resolve would enable it in the middle of the DAG (enabling driver chaining could be an option here too).
a
I think it will have to be with inject and resolve (cause I want to then add more nodes).
s
If it’s a known set of things, then you could just be verbose and use 
@config.when
over a set of functions
> I think it will have to be with inject and resolve (cause I want to then add more nodes). or you keep it as two separate DAGs? Right now things in the middle means a driver has to be reconstructed for a different set of values…
💡 we could think about enabling 
kwargs
only functions in the middle, that need values passed in via execute to know what they operate over…
e
So yeah, config.when is doable but a bit verbose. Inject/resolve are really powerful, but the API is a bit low level. One option is to define your own decorator that delegates to `inject`/`resolve`
Copy code
@inject_from_config(feature_set="features_to_use")
def put_features_together(**features: pa.Table):
     ...

def inject_from_config(feature_set: str):
    @functools.wraps(fn)
    def decorator(fn):
        fn = resolve(when=ResolveAt.CONFIG_AVAILABLE, decorate_with=lambda feature_sets: feature_sets[feature_set])
        return fn
    return decorator

dr = driver.Driver({"feature_sets" : {"features_to_use" : ["feature_table_1", "feature_table_2"]}})
dr.execute(["put_features_together"])
Note that this involves a limitation where 
resolve
refers to config items by name of parameter — this is something I’d want to change at some point, to make it easier so they don’t have to be nested under a fixed 
feature_sets
. That said, the high-level is that the code is easier to read/understand, and you can mess with the way it works under the hood…
a
OK, I’m not sure it has to be that complicated. The way I see it 
config={"requested_features": ["feature_table_1", "feature_table_2"}
. I’ll change the config between runs.
👍 1
🥵 got it to work, it all makes sense now but decorators require the brain to be switched on.
Copy code
@resolve(
    when=ResolveAt.CONFIG_AVAILABLE,
    decorate_with=lambda feature_tables: inject(
        feature_tables=group(
            **{feature_table: source(feature_table) for feature_table in feature_tables}
        )
    ),
)
def get_features(feature_tables: dict[str, pa.Table]) -> list[pa.Table]:
    return list(feature_tables.values())
🙌 2
btw, very helpful error messages.
s
> btw, very helpful error messages. thanks, we try to anticipate where things might break if we can.
e
Ahh yep, I think we can make this all easier to manage/simpler, glad the error message guided you :)
2 Views
Open in Slack
PreviousNext
Powered by