So, when you call out to

get_patterns

in

get_filepaths

, instead, you want

get_filepaths

to depend on

get_patterns

E.G.

def filepaths(patterns: List[str]) -> List:
    """returns all filepaths that match pattern"""
    paths = get_matching_paths(patterns)
    return paths

@config.when(data="cost")
def patterns__cost(org: int, provider: str, lookback: int) -> List[str]:
    """Retruns patterns that match for cost data"""
    patterns = []
    # some patterns logic...
    return patterns


@config.when(data="utilisation")
def patterns__util(org: int, provider: str, lookback: int) -> List[str]:
    """Retruns patterns that match for utilisation data"""
    patterns = []
    # some patterns logic...
    return patterns

and

load_data

should be something like this:

def data_loaded(spark: SparkSession, filepaths: List[str]):
    ...

So basically hamilton controls the DAG but you define the function — that way, you never have to wire through your inputs to the function!

Note I’ve also made a change to the way you name them that should make it easier to follow — I suggest using nouns to name your functions, rather than verbs. The trick to know is that each function corresponds to a set of artifacts, and your definition of them both tells you how they should be computed and what artifact they create.

Also the specific reason that

get_patterns

wasn’t being recognized is cause Hamilton doesn’t actually rename the function — it just sets the name of the nodes it creates. So, the functions were still called

get_patterns__util

, and when you called

get_patterns

python couldn’t find them 🙂