Hey folks, qq re the caching process. Per the docs

node_name: name of the node
code_version: version of the node's code
dependencies_data_versions: data_version of each dependency of the node

question is: are node input values taken from

.with_config()

considered part of dependencies_data_versions, or are only results from other nodes included? I've got a problem wherein I pull some data from a db and cache it, but I pull data based on a

max_date

value supplied through the config. If the cache is clean, then I get different results, for 3 different dates. On a second run though all results are the same, as if only the one of them is returned. Cache logs show cash hit all the time So I'm trying to figure out is it how caching works and I need to have a node with a date, so it's cached separately, or the problem is elsewhere

What is your Hamilton version? (as displayed via

pip list

)

Is it possible to share a picture of the DAG? To make sure I understand, you have something like

def data_from_db(max_date: datetime.datetime) -> pd.DataFrame: ...

def transform(data_from_db: pd.DataFrame) -> pd.DataFrame: ...

# 
dr = driver.Builder().with_config(max_date=datetime.datetime(...)).with_cache()
dr.execute()

I can share a pic if that helps

FYI, we previously fixed a bug for

datetime

objects. These classes are oddly defined compared to the rest of the Python standard library. It was a quick fix once we could pin point the issue!

ha, funny you should mention that, because that is exactly how I supply the date into the config

at the end of November, so you should have the fix in

1.85.1

If you can open a new issue with a small repro or reproduce yourself the

Investigation

section I shared here: https://github.com/DAGWorks-Inc/hamilton/issues/1242

Just checked. with the same example as in the issue, everything seems fine: datetime is correctly picked up by the data versioning and on the subsequent runs all works as expected. the problem is elsewhere in my code.

Great, that rules one issue out. My other hypothesis is related to config and input values (as you suggested). I'll investigate

I just tried a custom notebook and it did not trigger an issue for me there, I mean with the config. I also use a constant hasher for a monogdb client, tried to reporduce that and it did not work

There are a lot of things to unpack for me given I don't have access to the code (I don't see how the

max_date

relates to the

hash_db_client_object()

. With access to code I would do: 1. start with a clean cache and reproduce the bug you described (run with 3 different dates, expecting different result, but getting the same result) 2. access

Driver.cache

to get

.cache_keys

. You should get a dictionary of

{run_id: {node_name: cache_key}

3. Use

hamilton.caching.cache_key.decode_key()

on the cache key for

max_date

. You will get a dictionary of

{node_name, code_version, data_version}

4. Given your description, I expect that the

data_version

for the 3 different values of

max_date

are the same while you expect them to be different.

Another idea: I only request the last node in the graph that does not depend on the config. I wonder if that is somehow not taken into account when caching logic is performed. Anyway, I have no choice but to investigate here. When I find out I will report back. Hopefully it's something silly...

you can do

dr.cache.view_run()

to display a graph of what was retrieved from cache or not btw

I will investigate yeah. The max date is shown correctly in 3 different runs, all as expected, but i think what I saw is that the data versions for internal intermediate dataframes was the same