Is possible to use

parameterize

to define a new node in a file different from the original funcition? For example, I have file a.py:

def a(input1:int) -> int:
  ...

And a file b.py:

b = parameterize(b={"input1":value(1))(a)

When I try this, and import in the module init.py both a and b, I get "Cannot define function a more than once." when building the driver.

🤔 I don’t think we’ve had this request before. Which function does it complain about?

The full stack trace (the name of the modules is different, but is the same ideia of creating a new node using parameterize in another file):

ValueError                                Traceback (most recent call last)
Cell In[3], line 7
      1 tracker = adapters.HamiltonTracker(
      2     project_id=1,
      3     username="EltonCN",
      4     dag_name="toy1d_train"
      5 )
----> 7 d = driver.Builder().with_modules(toy1d, shared).with_adapter(tracker).build()
      9 d.display_all_functions(deduplicate_inputs=True)

File c:\Users\eltsu\AppData\Local\Programs\Python\Python312\Lib\site-packages\hamilton\driver.py:2163, in Builder.build(self)
   2156     grouping_strategy = self.grouping_strategy or grouping.GroupByRepeatableBlocks()
   2157     graph_executor = TaskBasedGraphExecutor(
   2158         execution_manager=execution_manager,
   2159         grouping_strategy=grouping_strategy,
   2160         adapter=lifecycle_base.LifecycleAdapterSet(*adapter),
   2161     )
-> 2163 return Driver(
   2164     self.config,
   2165     *self.modules,
   2166     adapter=adapter,
   2167     _materializers=self.materializers,
   2168     _graph_executor=graph_executor,
   2169     _use_legacy_adapter=False,
   2170     allow_module_overrides=self._allow_module_overrides,
   2171 )

File c:\Users\eltsu\AppData\Local\Programs\Python\Python312\Lib\site-packages\hamilton\driver.py:466, in Driver.__init__(self, config, adapter, allow_module_overrides, _materializers, _graph_executor, _use_legacy_adapter, *modules)
    464     error = telemetry.sanitize_error(*sys.exc_info())
    465     logger.error(SLACK_ERROR_MESSAGE)
--> 466     raise e
    467 finally:
    468     # TODO -- update this to use the lifecycle methods
    469     self.capture_constructor_telemetry(error, modules, config, adapter)

File c:\Users\eltsu\AppData\Local\Programs\Python\Python312\Lib\site-packages\hamilton\driver.py:440, in Driver.__init__(self, config, adapter, allow_module_overrides, _materializers, _graph_executor, _use_legacy_adapter, *modules)
    438 self.graph_modules = modules
    439 try:
--> 440     self.graph = graph.FunctionGraph.from_modules(
    441         *modules,
    442         config=config,
    443         adapter=adapter,
    444         allow_module_overrides=allow_module_overrides,
    445     )
    446     if _materializers:
    447         materializer_factories, extractor_factories = self._process_materializers(
    448             _materializers
    449         )

File c:\Users\eltsu\AppData\Local\Programs\Python\Python312\Lib\site-packages\hamilton\graph.py:736, in FunctionGraph.from_modules(config, adapter, allow_module_overrides, *modules)
    717 @staticmethod
    718 def from_modules(
    719     *modules: ModuleType,
   (...)
    722     allow_module_overrides: bool = False,
    723 ):
    724     """Initializes a function graph from the specified modules. Note that this was the old
    725     way we constructed FunctionGraph -- this is not a public-facing API, so we replaced it
    726     with a constructor that takes in nodes directly. If you hacked in something using
   (...)
    733     :return: a function graph.
    734     """
--> 736     nodes = create_function_graph(
    737         *modules, config=config, adapter=adapter, allow_module_overrides=allow_module_overrides
    738     )
    739     return FunctionGraph(nodes, config, adapter)

File c:\Users\eltsu\AppData\Local\Programs\Python\Python312\Lib\site-packages\hamilton\graph.py:175, in create_function_graph(config, adapter, fg, allow_module_overrides, *modules)
    173             continue  # This makes sure we overwrite things if they're in the config...
    174         if n.name in nodes and not allow_module_overrides:
--> 175             raise ValueError(
    176                 f"Cannot define function {n.name} more than once."
    177                 f" Already defined by function {f}"
    178                 f" In case you want to override the previous functions check out .allow_module_overrides() at: <https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/>"
    179             )
    180         nodes[n.name] = n
    181 # add dependencies -- now that all nodes except input nodes, we just run through edges & validate graph.

ValueError: Cannot define function toy1d_data more than once. Already defined by function <function toy1d_data at 0x000001C2F1B06CA0> In case you want to override the previous functions check out .allow_module_overrides() at: <https://hamilton.dagworks.io/en/latest/reference/drivers/Driver/>

is

toy1d_data

a

or

b

in your psuedo code above?

toy1d_data is a, and b is the new function "test_p" in another file

hmm @Elton Cardoso Do Nascimento this seems to work for me: # base.py

def a(input: int) -> int:
    return input * 2

# base_extended.py

from hamilton.function_modifiers import parameterize, value

import base

b_p = parameterize(
    b={"input":value(1)},  # this is what you can request to compute
    c={"input":value(2)}.  # this is what you can request to compute
)(base.a)

#base_run.py — this needs to import both modules

from hamilton import driver
import base_extended
import base
dr = driver.Builder().with_modules(base, base_extended).build()
dr.display_all_functions("base_extended.png")
print(dr.execute(['b', 'c']))

Yeah, this works. But, when I have something like:

my_module
  __init__.py
  base.py
  base_extended.py

Where the

___init___.py is

from .base import a
from .base_extended import b_p

and I pass

my_module

to the driver builder, it throws the exception

> (also, in base_extended.py, I imported a with

from .base import a

) yes so try doing

import base

and then referencing

base.a

Ok, but I cannot import base in base_extended, it throws a ModuleNotFoundError. It would only work if I calling the base_run from inside the my_module folder.

@Elton Cardoso Do Nascimento I got the following to work - but it does mean the

a

by itself isn’t available - not sure if this stop gap measure would work for you for now. Under: my_module/

__init__.py

from .base import z # key is to not import `a` -- if that happens we have redefinition error (something we could fix I think but this gets around it)
from .base_extended import b_p, d

base.py

def a(input: int) -> int:
    return input * 2

def z(input: int) -> int:
    return input * 3

base_extended.py

from hamilton.function_modifiers import parameterize, value

from my_module import base

b_p = parameterize(
    b={"input":value(1)},
    c={"input":value(2)} 
)(base.a) 

def d(b: int, c: int) -> int:
    return b + c

Then in my

run.py

outside

my_module

.

from hamilton import driver
import my_module
dr = driver.Builder().with_modules(my_module).build()
dr.display_all_functions("base_extended.png")
print(dr.execute(['b', 'c']))

Right, my ideia is to create a

shared

module with some base functions that I can use in my dataflows, and each specific experiment/project module can parameterize the shared function to create its node. So my problem here is that I don't want the node definitions of each dataflow to be coupled with the function definition itself, as that would reverse the direction of dependency.

Got it — yeah, the multiple parameterizations is a good use-case for this. The problem being that the decorator attaches to the function so it’s technically a one-time-use thing (this is by design, but doesn’t have to be). Will consult with @Stefan Krawczyk!

@Elton Cardoso Do Nascimento thanks! > So, ok, I thing that it would be nice to be able to use both “a” and the parameterization of “a”, but its not essential. Yeah so I think we should be able to support both. > But, another new problem that I just checked is that I cannot create two parameterizations in different places yep so you want to be able to do this, right? As yeah right now this would not work:

one_p = parameterize(
    b={"input":value(1)},
    c={"input":value(2)} 
)(base.a) 

two_p = parameterize(
    d={"input":value(3)},
    e={"input":value(4)} 
)(base.a)

Feel free to add any other shapes/patterns to https://github.com/DAGWorks-Inc/hamilton/issues/1281

Yes