Hello I am wanting to use ` with materializers ` to save out Hamilton Open Source #hamilton-help

Hello, I am wanting to use `.with_materializers()`...

Seth Stokes

02/10/2025, 9:12 PM

Hello, I am wanting to use

.with_materializers()

to save out intermediate step in a dag. I am currently trying the following but am encountering some issues. Details in thread.

Seth Stokes

02/10/2025, 9:14 PM

Code:

Copy code

from hamilton import driver, base
from hamilton.io.materialization import to

ITEMS = []
ALL_DFS = []
for item in ITEMS:
    builder = (
    driver.Builder()
    .with_modules(__main__)
    .with_adapters(
        base.PandasDataFrameResult(), 
        # lifecycle.PDBDebugger(
        #     "c.with_summarize_breaks",
        #     during=True
        # )
    )
)
    builder_copy = builder.copy()
    materializers = [
        to.excel(id="debug_a", dependencies=["a"], path=f"{item}_debug_a.xlsx"),
        to.excel(id="debug_b", dependencies=["a.with_filter_out_noise"], path=f"{item}_debug_a_with_filter_out_noise.xlsx"),
        to.excel(id="debug_c", dependencies=["b"], path=f"{item}_debug_b.xlsx"),
        to.excel(id="debug_d", dependencies=["b.with_summarize_breaks"], path=f"{item}_b_with_summarize_breaks.xlsx"),
    ]
    dr = builder_copy.with_config({"child_rec": item}).with_materializers(*materializers).build()
    df = dr.execute(
        ["c", *[f"debug_{x}" for x in list("abcd")]], 
        inputs={"save_location": save_location}
    )
    ALL_DFS.append(df)

Seth Stokes

02/10/2025, 9:17 PM

I would like to get out a

DataFrame

but I think including the

debug_x

nodes is causing an issue building the result.

Copy code

df = dr.execute(["c", *[f"debug_{x}" for x in list("abcd")]], inputs={"save_location": save_location})
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\driver.py", line 637, in execute
    raise e
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\driver.py", line 628, in execute
    outputs = self.adapter.call_lifecycle_method_sync(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\lifecycle\base.py", line 955, in call_lifecycle_method_sync
    return getattr(adapter, method_name)(**kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\lifecycle\api.py", line 64, in do_build_result
    return self.build_result(**outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\base.py", line 225, in build_result
    return PandasDataFrameResult.build_dataframe_with_dataframes(outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\hamilton\base.py", line 278, in build_dataframe_with_dataframes
    return pd.DataFrame(flattened_outputs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "c:\codebase\00-tidal\api-duco\duco-venv\Lib\site-packages\pandas\core\frame.py", line 778, in __init__
    mgr = dict_to_mgr(data, index, columns, dtype=dtype, copy=copy, typ=manager)

...

TypeError: '<' not supported between instances of 'str' and 'int'
    indexer = non_nan_idx[non_nans.argsort(kind=kind)]
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
TypeError: '<' not supported between instances of 'str' and 'int'

Seth Stokes

02/10/2025, 9:29 PM

Solution was to remove

base.PandasDataFrameResult()

and:

Copy code

df, *_ = dr.execute(
    ["c", *[f"debug_{x}" for x in list("abcd")]], 
    inputs={"save_location": save_location}
).values()

👍 2

Seth Stokes

02/11/2025, 2:52 AM

If I am using

pipe_input

with only one

step

would asking for node

and node

a.with_step1

return the same result ?

2 Views

Open in Slack

Previous Next