good morning Lets say I have a dag that has 4 data loading n Hamilton Open Source #hamilton-help

good morning. Lets say I have a dag that has 4 dat...

Seth Stokes

12/02/2025, 2:05 PM

good morning. Lets say I have a dag that has 4 data loading nodes. my goal is recompute all nodes that DO NOT load data as the default behavior. I would like to avoid explicitly defining the default nodes themselves what is the best approach to accomplish this? Below is my attempt but it seems to still be recomputing. Am I missing something?

Copy code

driver.Builder().with_cache(
    ignore=[
        "data_source_1",
        "data_source_2",
        "data_source_3",
        "data_source_4",
    ], default_behavior="recompute"
)

Seth Stokes

12/03/2025, 8:48 PM

Additionally, would you advise to inspect intermediate nodes from the

cache

or by asking for each one from the

driver

? Is the cache less memory intensive or is there not much difference?

Stefan Krawczyk

12/04/2025, 12:51 AM

https://hamilton.apache.org/reference/caching/caching-logic/#hamilton.caching.adapter.CachingBehavior

Stefan Krawczyk

12/04/2025, 12:52 AM

if you want to recompute all nodes, without any data changing, then I think you need to mark them to be explicitly recomputed.

Stefan Krawczyk

12/04/2025, 12:52 AM

with caching the default is to not recompute

Stefan Krawczyk

12/04/2025, 7:32 AM

otherwise if you made code changes, then only those nodes should recompute and things downstream of it...

👍 1

Seth Stokes

12/10/2025, 5:47 PM

Copy code

.with_cache(recompute=["data_source_1"])

I have been toggling on/off recompute which has yielded the desired behavior. 1. Ask for one terminal node via

.execute()

2. Debug from cache if necessary (instead of predefining execute nodes). 3. Rerun

.with_cache()

4. Recompute

.with_cache(recompute=[...])

when fresh data is desired.

Seth Stokes

12/10/2025, 5:52 PM

~~The problem I now face is that is that (4) still recomputes..~~ Disregard.. I just realized that this is due to an un-serializable

pd.Timestamp

. So I guess the solution is to convert to

datetime

Seth Stokes

12/17/2025, 5:12 PM

When using

@pipe_output

there becomes a new node

<node>.raw

so when using

.with_cache(recompute=["data_source_1"])

it was not exactly clear why my

data_source_1

was not recomputing despite seeing data_source_1::adapter::execute_node
in the logs. Is this the behavior you would like? Of course

dr.cache.view_run()

enabled me to figure this out in the end so perhaps everything is already as it should be.

Copy code

# transforms.py
@pipe_output(step(filter_to_usd))
def data_source_1() -> pd.DataFrame:
    ...

def data_source_2() -> pd.DataFrame:
    ...

def reference_data(data_source_1: pd.DataFrame) -> pd.DataFrame:
    ...

def final_df(data_source_1: pd.DataFrame, data_source_2: pd.DataFrame, reference_data: pd.DataFrame) -> pd.DataFrame:
    ...


# main.py (with hidden perhaps behavior)

dr = driver.Builder().with_cache(recompute=["data_source_1"]).build()
result = dr.execute(["final_df"])
# >>> data_source_1::adapter::execute_node

Copy code

# main.py (with expected behavior)
dr = driver.Builder().with_cache(recompute=["data_source_1.raw"]).build()
result = dr.execute(["final_df"])
# >>> data_source_1.raw::adapter::execute_node
# >>> database logs

I am also seeing that for some reason, a cached node with and upstream recomputed node is pulling from the cache instead of being recomputed... so in this example

final_df

returns from cache despite setting

recompute=[data_source_1.raw]

Stefan Krawczyk

12/18/2025, 12:26 PM

is it possible that upstream yields identical results?

Open in Slack

Previous Next