Morning should this be equivalent ``` extract columns COLS T Hamilton Open Source #hamilton-help

Morning, should this be equivalent? ```@extract_co...

Seth Stokes

03/26/2024, 4:02 PM

Morning, should this be equivalent?

Copy code

@extract_columns(
    *COLS_TO_EXPOSE
)
@pipe(
    step(_filter_to_src_trades),
    step(_filter_to_unrealized)
)
@config.when(env="test")
def raw_data_src__test(
    cob_date: datetime, 
    cache_dir: str = r"C:\codebase\data"
    ) -> pd.DataFrame:
    return query(cob_date)

Copy code

@extract_columns(
    *COLS_TO_EXPOSE
)
@config.when(env="test")
def raw_data_src__test(
    cob_date: datetime, 
    cache_dir: str = r"C:\codebase\data"
    ) -> pd.DataFrame:
    raw = query(cob_date)
    return raw.pipe(_filter_to_src_trades).pipe(_filter_to_unrealized)

Asking because of the following error.

Copy code

ValueError: Error: raw_data_src is expecting raw_data_src.with_filter_to_unrealized:<class 'datetime.datetime'>, but found raw_data_src.with_filter_to_unrealized:<class 'pandas.core.frame.DataFrame'>. 
Hamilton does not consider these types to be equivalent. If you believe they are equivalent, please reach out to the developers. Note that, if you have types that are equivalent for your purposes, you can create a graph adapter that checks the types against each other in a more lenient manner

Seth Stokes

03/26/2024, 4:05 PM

By the looks of it,

hamilton pipe

is intended to perform on the input parameter values and not the return value?

Stefan Krawczyk

03/26/2024, 4:06 PM

@Seth Stokes the

@pipe

is run before the function. So no, they aren’t equivalent.

Stefan Krawczyk

03/26/2024, 4:06 PM

We have an issue open to create

@post_pipe

which would mirror the behavior you’re expecting

Stefan Krawczyk

03/26/2024, 4:08 PM

So you’ll want to change things to be like this:

Copy code

@config.when(env="test")
def raw_data_src__test(
    cob_date: datetime, 
    cache_dir: str = r"C:\codebase\data"
    ) -> pd.DataFrame:
    return query(cob_date)

@extract_columns(
    *COLS_TO_EXPOSE
)
@pipe(
    step(_filter_to_src_trades),
    step(_filter_to_unrealized)
)
def processed_data(raw_data_src: pd.DataFrame) -> pd.DataFrame:
    return raw_data_src # transforms in pipe have been applied

Seth Stokes

03/26/2024, 4:17 PM

beautiful, ty sir - assuming it’s this one https://github.com/DAGWorks-Inc/hamilton/issues/701

Open in Slack

Previous Next