Hi there 🙂 I would like to add different node highlighting to the DAG based on the state of the cache of the diskcache_adapter to achieve something similar to the visualisations of the

targets

R package (https://books.ropensci.org/targets/walkthrough.html#change-code). Is there a way of querying the driver for differences between it's cache and workflow? Something like a dry-run that checks the nodes but doesn't execute them.

I'm on my phone, so I'd have to look at the source code. But technically that should be possible, but my guess is that it might not be exposed.

Production and development, though "production" means something slightly different in my case. I'm primarily a research scientist, so production is more in the sense of publishing an analysis, for which it is nice to have a visual confirmation that the output of a workflow is up-to-date. And for development I like to periodically check my workflow for "If I ran this now, which nodes would be recomputed based on the cache and changes to the code?" before e.g. deciding if this is a run I start before a coffee or just a small update I execute immediately.

Cool. Are you developing in an notebook? or IDE? or running through a CLI?

And if this works I would ask if you are interested in a PR as part of the driver visualization

So something like https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/cli#diff in terms of output? (for context, this does a diff between git commits on a module)

I'm working in Neovim, where I have workflows as python modules and import those into a quarto notebook (quarto is like plain text jupyter notebooks) from which I interactively run code in an ipython console or execute/render the whole notebook to various output formats (works out of the box with hamilton so far!)

yup, so the diff viz here is based on the node's code version. Now, we'd have to create a mechanism to consider the "data version" used internally by the diskcache feature. Just to be sure, you'd want to see the visualization before execution the dataflow?

The way we have it right now, would it make sense to have a viz that distinguishes: • the inputs you pass • the nodes we're certain we can read from cache • the nodes we might have to compute (don't know until execution) • the nodes we must compute It might be a bit difficult to know 100% what can be read from cache before execution

@Jannik Buhr what is the precedence that you’d expect? 1. Write code. 2. Run - things are cached. 3. Modify code of node 4. Run - only modified node & downstream of it is rerun? 5. Is this invariant to inputs being changed? or?

oh, btw. I don't want to come across as that "guy who has just discovered someones framework and now wants to make it work like this other framework he likes" :D

No worries! You have good timing because we're currently looking at overhauling the cache features haha

Feel free to tell me if I'm completely missing a point or something :)

heh no worries - we’re trying to figure out the best path forward for “caching” and this is great input. E.g. is it check pointing (invariant of code & inputs)? is it some sort of intelligent caching? (change code, rerun things only downstream of it, change input, rerun only what’s downstream) or something in between how do people want to use it? etc

In that case taking inspiration from the ux of targets might actually be helpful, because this works incredibly well for many for those data science tasks. I'll try to summarize key features

Looking at the code - we have the individual bits, but not wired together in the same place right now. • driver visualization path doesn’t have access to adapters — so nothing quick to be done there. • disk cache knows what nodes it has a cache for and their code hash versions — this could be pulled out from the adapter, e.g. we could expose a function, what do I have and what code versions? • We have the building blocks to provide node hashes given a driver. • We then can figure out the path between nodes and outputs So right now you could probably code up a utility function that does the following:

def hash_hamilton_nodes(dr: driver.Driver) -> Dict[str, str]:
    """Hash the source code of Hamilton functions from nodes in a Driver"""
    from hamilton import graph_types

    graph = graph_types.HamiltonGraph.from_graph(dr.graph)
    return {n.name: n.version for n in graph.nodes}

def what_is_still_valid_in_cache(dr: driver.Driver, disk_cache: ...) -> list[str]:
    node_hashes = hash_hamilton_nodes(dr)
    nodes_in_cache = disk_cache.nodes_history 
    result = []
    for node, node_versions in nodes_in_cache.items():
        # caveat assumes we produce the same hashes -- would need to double check code paths here
        current_hash = node_hashes[node]
        if current_hash in node_versions:
            result.append(node)
    return result

# then you could minimally style the viz that way
# see <https://github.com/DAGWorks-Inc/hamilton/tree/main/examples/styling_visualization>

ideally we’d figure out the paths impacted, but at least visually you could see something quickly…