https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
can subdag be combined with pipe on the same node,...
# hamilton-help
v
can subdag be combined with pipe on the same node, to ingest the file data via subdag and preprocess it via pipe in one go?
when I add pipe, I get the error:
Copy code
InvalidDecoratorException: Function: raw_urls_multiline has a first parameter that is not a dependency. @pipe requires the parameter names to match the function parameters. Thus it might not be compatible with some other decorators
This is my code
Copy code
@subdag(
    textwidget_store,
    inputs = {"textwidget_location": source("important_urls_location")},
    config = {}
)
@pipe(
    step(_multiline_cleaning)
)
def raw_urls_multiline(textwidget_load: str) -> str:
    ...
Is this the incompatibility mentioned in the error message?
e
So I think that’s not supported… Although it definitely is an intersting case.. Just to be clear, you want to: 1. have 
subag
run 2. Pass in the result of a 
pipe
command from subdag The problem is that the subdag parameter gets passed from the resulting DAG. 
pipe
also expects to pass it in so there’s a conflict. I’d make the pipe part of the subdag, which (IMO) clarifies it a bit.
👍 1
v
that makes sense
Eventually, I'll need to think about possibly making a part of the early pipeline nodes a DataSaver / DataLoader. This is good for practicing, though.
👍 1
Open in Slack
PreviousNext
Powered by