Hi, does Hamilton support the following: if an input is missing, automatically skip executing the subdag that depends on it. The use case is to implement backward compatibility when adding a new input

new_df

. It would be useful to be still able to run the DAG on an older dataset that doesn’t have

new_df

.

🤔 we don’t have that explicit functionality. Some thoughts: • Functions can take default values. So if an input is missing, it has a default value. • @config.when could be useful, and you could resolve it at DAG build time via @resolve. • we have the graceful error adapter that maybe could be used here? • else you could create two DAGs (one for new, one for old) and then know which one to call? • or use hamilton within hamilton

@Yijun Tang If I understand, you're managing a project where you have a

version 1

that's being used, you made some change to create

version 2

, and now both versions are in use. Sounds accurate? I think @config.when can be useful. This allows you to have 2 nodes with the same name but different implementation. Let's say your

version 1

looked like this:

# report.py
import pandas as pd

def monthly_report(sales: pd.DataFrame) -> pd.DataFrame:
   return ...

# where you execute code
import report
from hamilton import driver

dr = driver.Builder().with_modules(report).build()
dr.execute(...)

Then, as your project evolve and you want to modify

monthly_report()

you could use

@config.when

# report.py
import pandas as pd
from hamilton.function_modifiers import config

@config.when_not(version="v2")
def monthly_report__v1(sales: pd.DataFrame) -> pd.DataFrame:
   return ...

@config.when(version="v2)
def monthy_report__v2(sales: pd.DataFrame, new_df: pd.DataFrame) -> pd.DataFrame:
  return ...

Now, when you create your

Driver

you can pass a config. By passing

version="v2"

in a dictionary it will use the implementation that has

new_df

. If the config is empty, it will use the

v1

implementation

import report
from hamilton import driver

dr = (
  driver.Builder()
  .with_modules(report)
  .with_config({"version": "v2"})
  .build()
)
dr.execute(...)

Thank you both! I’m trying to implement the following. The gray nodes are V1. The red nodes are added in V2. We’d like to run just the gray nodes on V1 dataset.

Check out the docs. You only need to annotate the functions that are different. IIUC.

Ah ok, seems like I can mark the gray nodes with

@config.when_in(param=["v1", "v2"])

and the red ones with

@config.when_in(param=["v2"])

. This way when V3 is introduced, we can add

"v3"

to both lists.

to clarify, are successive versions purely additive? or do they actually replace certain functions (i.e. nodes)? or both?

Great question, for now I think they will be additive in our use case, and perhaps deprecating some old nodes, instead of replacing nodes.

Without knowing more about how you’re determining what to “request” as output there’s a few more possibilities.

Ah thanks!

GracefulErrorAdaptor

is looking very promising. I’ll give it a try.