Hello Trying out caching functionality with a simple set up we have so far and noticing that it might not be working as I would expect We split out the result of

legacy_cashflow_computation

into 3 sub results using

@extract_fields

, it looks like all of them are being picked up from cache correctly, however it seems like we still re-run

legacy_cashflow_computation

I understand that the reason we are re-running right now is because we have not handled

pd.DataFrame

and

pydantic.BaseModel

hashing properly using

fingerprinting

yet However, I would have expected it to understand that given we picked up all 3 outputs of a node from cache, it should stop traversing the graph further. Am I understanding the intended behaviour incorrectly or could this be a bug? Appreciate any help on this!

@Vladyslav Bondarenko thanks for the question. Caching right now doesn't behave in a checkpointing manner. We have to evaluate every node to see if anything changed -- this is mainly to enable you to change an input and have things propagate, whereas with checkpointing, it would continue from where it could. Which behavior are you more interested in? CC @Thierry Jean

Hi! I suggest you go through the caching tutorial. The later sections should provide you the tools to debug and inspect the cache. From the info I have, nothing indicates a bug ## How it works Caching is complex and we all approach with different assumptions. Without seeing your data and code, it's hard for me to explain exactly what's going on. Here's what I can infer: • if

opex_growth_detail

is retrieved from cache; then, I know ◦ the code for

opex_growth_detail

was previously run (fact 1) ◦ with the same upstream values (it only depends on

legacy_cashflow_computation

fact 2) • if

legacy_result_ordered

is retreived from cache; then, I know ◦ the code for

legacy_result_ordered

was previously run (fact 1) ◦ with the same upstream values (it only depends on `legacy_cashflow_computation`; fact 2) • etc. ## The surprise Now, regarding the "surprising result", why is

legacy_cashflow_computation

executed? • fact 1: did the same code run previously? • fact 2: with the upstream values

(project, yearly_inflation_rpi, opex_costs, yearly_inflation, cpi_rpi_indexes, custom_growth_rate_overwrite, tenants_df, multiyear_assumptions, user_context, cashflow_growth_profile_mapping)

◦ this requires having stable fingerprints for all these classes objects ## Debugging The inputs to

legacy_cashflow_computation

seems very complex. See the end of the tutorial notebook to learn how to inspect your cache keys. You should be able to decode something like this:

{
  'node_name': 'legacy_cashflow_computation',
  'code_version': 'c2ccafa54280fbc969870b6baa445211277d7e8cfa98a0821836c175603ffda2',
  'dependencies_data_versions': {
    'project': 'WgV5-4SfdKTfUY66x-msj_xXsKNPNTP2guRhfw==',
    'yearly_inflation_rpi': ...,
    ...,
  }
}

This will allow you to pin point what

data_version

is causing the cache misses. Then, you can fix your inputs to be stable (order matters for

opex_costs

since it's a `list`; if order doesn't matter, use a

set

) or implement a custom fingerprinting function

Thanks @Stefan Krawczyk @Thierry Jean! @Stefan Krawczyk - The caching would still be very useful for us, though I wonder about the overhead when scaling to larger graphs. With 100s of nodes doing relatively simple computations on large arrays, could the evaluation overhead become significant compared to the computation time itself? @Thierry Jean - Found it! Used the cache inspection tools and you were right, one of our inputs was incrementing a counter on every run facepalm. Also nice that pandas/pydantic hashing just works out of the box 👍 Appreciate the help!

@Vladyslav Bondarenko yep there is work to optimize things -- ideally we should only be checking stored hashes which should be fast... and there's also ways to disable caching for nodes / parts of the graph...

This tutorial notebook shows how you can load the entire result store or metadata store in-memory to limit round-trip to disks. It's a trade-off between RAM and I/O operations. Larger improvements to the caching algorithm will likely have to wait for Hamilton 2.0 and could involve some breaking changes