Afternoon sirs, is there a way i can get all of this at once? Assuming each

hist_data

is a normalized dataframe. I’d like this node to return an appended dataframe from each source.

@parameterized_subdag(
    file_path, raw_data_src1, hist_data,
    from_datasource_1={"inputs" : {"file_paths" : source("INPUT_FILES_DATASOURCE_1")}},
    from_datasource_2={"inputs" : {"file_paths" : source("INPUT_FILES_DATASOURCE_2")}},
    from_datasource_3={"inputs" : {"file_paths" : source("INPUT_FILES_DATASOURCE_3")}}
)
def etl_pipeline(hist_data: pd.DataFrame) -> pd.DataFrame:
    return hist_data

Slack Conversation

That code creates three sub-dags that return a dataframe as a result. IIUC, you want to combine the results of each one? If so, for that I’d just add one more function:

def all_datasources(from_datasource_1: pd.DataFrame, ...) -> pd.DataFrame:
   return pd.concat([from_datasource_1, ...])

Ok sweet, I’ll try that. I think this enables me to have multiple sync uses of Parallelizable/Collect in the same dag. Thanks!