Seth Stokes
04/23/2024, 2:41 PMpipe
filter step
to a node, could this with_filter_criteria
node be accessed from the driver?
E.g.
def _filter_criteria(df: pd.DataFrame, col: str) -> pd.DataFrame:
return df[df[col] == "US"]
@pipe(step(_filter_criteria, col="region"))
def filtered_data(raw_data: pd.DataFrame) -> pd.DataFrame:
# some work with raw_data
Pardon.... just had to change
dr.execute(["with_filter_criteria"])
to
dr.execute(["filtered_data.with_filter_criteria"])
Seth Stokes
04/23/2024, 2:52 PMpipe
filter steps automatically reset the index?
I know previously when having intermediate filtered nodes, that if i didnt include df.reset_index(drop=True)
it would cause issues.Elijah Ben Izzy
04/23/2024, 3:09 PM