https://github.com/stitchfix/hamilton logo
Join Slack
Powered by
Hi guys, qq re pairing dataloders and caching. So ...
# hamilton-help
k
Hi guys, qq re pairing dataloders and caching. So I have a data loader that reaches out to a mongodb and returns a tuple of a 
pl.DataFrame
and metadata, something like this
Copy code
@dataloader()  # type: ignore[misc]
def A(
    mongo_client: DocumentMongoClient,
) -> tuple[pl.DataFrame, dict[str, Any]]:
If caching on the driver is enabled with
Copy code
dr = (
    driver.Builder()
    .with_modules(temp_module)
    .with_modules(one_hot_module)
    .with_cache()
    .with_config({"stage": "training"})
    .build()
)
, is that stage supposed to cached and the database is not to be reached or does it work differently? The reason I’m asking is because I clearly see that the result of this node is depicted as 
from cache
on a 
dr.cache.view_run()
, but what I also see is that there is a massive spike in networking activity from my laptop when the graph is called. So like on the pic, the dataloders are not drawn as cached(they should not be I guess), and the result is cached, yet there is clearly data download happening every time this is called.
s
So looking at the “Materializers” section of this tutorial - https://github.com/DAGWorks-Inc/hamilton/blob/main/examples/caching/tutorial.ipynb It should be cached, i.e. it should become “red” outline with 
view_run()
.
Because it isn’t, my guess is that the hash for the 
mongo_client
might be the issue?
yep that seems to be the issue
@Konstantin Sofeikov to fix this (we need to add this to the docs) you need to provide a way to “hash” the db_client to a constant.
Copy code
from hamilton.caching import fingerprinting

from mongodb import DocumentMongoClient # I'm guessing this part

@fingerprinting.hash_value.register(DocumentMongoClient)
def hash_db_client_object(obj, *args, depth: int = 0, **kwargs) -> str:
    """Object hash"""
    return "constant" # change this obviously if you want to differentiate between clients, etc.
Do that where your driver code lives. Then this should make the db_client have a consistent hash so that then caching should work as expected.
💡 1
👍 1
k
@Stefan Krawczyk amazing, thank you
👍 1
Open in Slack
PreviousNext
Powered by