Hi - I have a use case where we define a set of inputs in an external config, then create extractors for that config at runtime. We reuse this across a number of Hamilton dags, but there are some instances where we wish to reuse these "standard" sets of extractors, but a dag does not include a reference node and we hit builder errors. Is there an existing way to handle optional inputs that a dag does not reference?

just so I understand you want to continue to use

from_.parquet

here? or in this instance, it's not used?

Yeah a minimal example would be super helpful to grok it!

Yes, the idea is that we have a shared set of materializers, but not all inputs may be used by all of the dags: Standard/reused job configs:

required_inputs:
  input1: parquet_path1
  input2: parquet_path2
optional_inputs:
  input3: parquet_path3

this config is being applied to multiple dags. dag_a:

def processed_data(input1, input2) -> pd.DataFrame:
    ...

def processed_optional(input3) -> pd.DataFrame:
    ....

def next_step(processed_data, processed_optional) -> pd.Dataframe:
    ....

dag_b is a "slim" version, and doesn't actually reference dag_input3, but uses the same inputs otherwise:

def processed_data(input1, input2) -> pd.DataFrame:
    ...

def next_step(processed_data) -> pd.Dataframe:
    ....

When we run these dags, we have

builder.with_module(dag_module)
materializers = [from_.parquet(target=target, path=path) for target, path in load_standard_materializers()]
builder.with_materializers(*materializers)
dr = builder.build()

This logic works fine if we load dag_a, but will result in build errors if we run dag_b:

dr = builder.build()
         ^^^^^^^^^^^^^^^
  File "hamilton/driver.py", line 2189, in build
    return Driver(
           ^^^^^^^
  File "hamilton/driver.py", line 492, in __init__
    raise e
  File "hamilton/driver.py", line 476, in __init__
    self.graph = materialization.modify_graph(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hamilton/io/materialization.py", line 486, in modify_graph
    loader_nodes.extend(loader.generate_nodes(fn_graph))
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "hamilton/io/materialization.py", line 166, in generate_nodes
    raise ValueError(
ValueError: Could not find node with name: input3 in function graph. Available nodes: ['input1', 'input2', Ellipsis]

At runtime, without building the dag first or doing module introspection, I'm not sure if there is an easy way to check that we don't use input3 and omit it from our materializers. Defining a unused input3 in the dag would also load unnecessary data.

Great question! There isn't a built-in way to do this today. Hacky workaround that could work: You can use inspect to gather parameter names from your DAG module and filter your materializers before passing them to the builder --- this only works if the target isn't supposed to feed into a decorator...

import inspect
 all_params = set()                                                                                                                                                                               
 for name, fn in inspect.getmembers(dag_module, inspect.isfunction):                                                                                                                              
     all_params.update(inspect.signature(fn).parameters.keys())                                                                                                                                   
  materializers = [from_.parquet(target=t, path=p) for t, p in load_standard_materializers() if t in all_params]

Regarding the feature contribution: Yes, we'd welcome a PR for an optional flag on from_ extractors. The change is in ExtractorFactory in hamilton/io/materialization.py — adding an optional parameter that makes generate_nodes return an empty list instead of raising when the target isn't in the graph. It's a clean, backward-compatible addition. If you open an issue we can discuss naming (

optional

vs

allow_missing_target

) — I slightly prefer

optional

for brevity since the context makes it clear what's optional.

from_.parquet(target="input3", path=some_path, optional=True)

sounds good, I will try out the workaround for now and create an issue to formalize this contribution. Thank you

Ok I think optional is reasonable, nice! I also think that the workaround isn't crazy but we should in theory be able to do it without doing inspect which I think is harder than it seems

Agreed - our weird use case is that we can use an external config, but plug in multiple dags (almost always a "ful" and "slim" version of a dag). This is definitely an edge case that we shouldn't hit often, but it came up recently, and the other way of handling this is to double these external configs and have dedicated ones for the slimmed down modules, which adds a ton of boilerplate and introduces opportunities for config drift in the future.