I m assuming one s input is someone else s output wink Hamilton Open Source #hamilton-help

Join Slack

I'm assuming one's input is someone else's output ...

# hamilton-help

Iliya R

08/27/2024, 3:44 PM

I'm assuming one's input is someone else's output 😉

Yijun Tang

08/27/2024, 4:01 PM

Haha that’s usually the case. However, I’m hoping to validate a dataframe input provided by materializer, loaded from parquet for example.

👍 1

Iliya R

08/27/2024, 4:02 PM

So why not validate the materializer output?

Iliya R

08/27/2024, 4:06 PM

A workaround that comes to mind is a passthrough task (i.e. output=input), after the materializer, that only does output checking

Stefan Krawczyk

08/27/2024, 4:13 PM

@Yijun Tang oh I see. I think you can combine the two. How are you using the materializer? As a decorator? or injecting it? or?

Stefan Krawczyk

08/27/2024, 4:19 PM

I think via the decorator we can combine them:

Copy code

@check_output(schema=...target_="a.select_data.input")
@load_from.csv(path=value('foo'))
def a(input: pd.DataFrame) -> pd.DataFrame:
    return ...

The above “compiled” but I didn’t run it. Otherwise for injecting I haven’t tested it, but you might be able to inject to the right node if you have this defined:

Copy code

@check_output(schema=...)
def a(input: pd.DataFrame) -> pd.DataFrame:
    return ...

and then you inject

a_raw

instead of

💡 1

Yijun Tang

08/27/2024, 5:59 PM

Thanks for the suggestion. I’m using materializers this way:

Copy code

materializers = [
        from_.parquet(target="raw_df_1", path=data_path),
        from_.parquet(target="raw_df_2", path=data_path),
    ]
    dr = driver.Builder().with_modules(__main__).with_materializers(*materializers).build()

Yijun Tang

08/27/2024, 6:01 PM

The reason I went this route is so that users don’t have to explicitly write the loader nodes

👍 1

Open in Slack

Previous Next