Otherwise, I've been thinking about a way to surface from the Driver all the: • Config keys • Required inputs • Optional inputs to automatically generate a sort of "configuration space" of all possible DAGs (without using overrides). Does that sound useful? It could help with ML experiments, or help DAG authors to expand/constraint the input space to only valid "configurations" for downstream users

@Thierry Jean I am using

_get_default_input

now indeed to check this but since this method is not expected to be used outside the module it might be good to make it clear that you can actually use this method (i.e. a publicly available method).

To illustrate the problem @Elijah Ben Izzy, when the following function is called:

def A(external: int = 7) -> float:
  return external / 2

# call 1 
dr.execute(["A"])
# call 2 
dr.execute(["A"], inputs={"external": 5})

For call 1 an call 2,

external

should be

is_external_input

(from the dataflow's POV) and they will have value 7 and 5 respectively. In the experiment tracker, I disambiguate the two using

inputs = dict()  # from .execute()
logged_inputs = []

# graph is graph_types.HamiltonGraph
# node is graph_types.HamiltonNode
for node in graph.nodes:

  # filter out config nodes which are external but no origin
  elif node.is_external_input and node.originating_functions:
    self.logged_inputs.append(
      dict(
        name=node.name,
        value=inputs.get(node.name),  # value from execute
        default_value=_get_default_input(node),  # value from function signature
      )
    )

which would lead to call 1

{name: A, value: 7, default_value: None}

and call 2

{name: A, value: 5, default_value: 7}

@Roel Bertens created https://github.com/DAGWorks-Inc/hamilton/issues/710 to track, feel free to comment

@Thierry Jean Hi, so I've been using this already to identify inputs with default values, which works. I use this now (feedback welcome):

def _is_input_node_with_default(node) -> bool:
    """Return True if the input node has a default value specified

    NOTE: also return True if the default is None
    """          
    if node.is_external_input and node.originating_functions is not None:  # exclude config nodes
        origin_function = node.originating_functions[0]
        param = inspect.signature(origin_function).parameters[node.name]
        return False if param.default is inspect._empty else True
    return False

Based on this I want to style these input nodes differently but I notice they are not reachable via the

custom_style

function which I pass to

display_all_functions

. How can I change their look?

where a and b should have a different style

wrt styling I am using somethng like this

def custom_style_function(
    *, node: graph_types.HamiltonNode, node_class: str
) -> Tuple[dict, Optional[str], Optional[str]]:
    if _is_input_node_with_default(node):
        style = ({"fillcolor": "#c1f5cf"}, node_class, "optional")
    else:
        style = ({}, node_class, None)

    return style

And then run

dr.display_all_functions(custom_style_function=custom_style_function))

There is also another thing that I was already considering which doesn't solve the same problem but still helps with creating clarity in the DAG. I would like a config setting to exclude the optional inputs from the DAG. Is that something useful to support or better to try and implement myself?

@ThreadScribe can you summarize this discussion into a structured GitHub issue for me?

Hey @Thierry Jean, your summary is ready! ### Clarifying Optional Inputs for Hamilton Driver In a recent discussion, the team explored ways to visually distinguish optional inputs from required ones in a Hamilton driver. Here are the key points from the conversation: - The request was to make it clear to the user that certain inputs to the driver are optional. This was in context of differentiating "optional" from "required" inputs in the visualizations. - A utility function was implemented in the

hamilton.plugins.h_experiments.ExperimentTracker

to get the default parameters of a node. This function inspects the function signature to distinguish default values for inputs. - The team also discussed the possibility of surfacing all the configuration keys, required inputs, and optional inputs to automatically generate a "configuration space" of all possible DAGs. This could potentially help with ML experiments or enable DAG authors to constrain the input space to valid configurations for downstream users. - Discussions included moving certain methods to different modules or properties for better accessibility and usage. - Sample code snippets were shared for identifying and styling input nodes with default values, along with the challenges and options for applying different styles to input nodes. - Short-term solutions were discussed, including support for ungrouping input nodes and creating new logic to identify default inputs for styling or hiding. - Graphviz limitations were highlighted, along with the potential challenges in implementing certain display options. - It was noted that excluding optional inputs from the DAG displayed a complex set of challenges and limitations, particularly in relation to Graphviz and the current API logic. The team also agreed to summarize the discussion into a structured GitHub issue to further track the progress of the potential changes. Feel free to refer to the complete discussion for additional context.

You can also view your Scribe HERE 👀