Hello, as I was discovering more about the library, I noticed that to create a Hamilton driver, I can use either

class hamilton.driver.Driver

or

class hamilton.driver.Builder

. What’s the difference and when to use each? Perhaps this was already answered, having the links would be appreciated. 🙂

Builder has more ergonomic capabilities for constructing DAGs— it creates a driver. Driver is largely there as it was what was released first. You should start with builder.

To add, the

Driver

object is a core object of Hamilton that you will be using all the time. Notably,

.execute()

executes your DAG. Now, what is the

Builder

/ why does it exist? In early versions, you would create the driver directly

dr = Driver(...)

But as the library became more complex and we added features, we added the

Builder

, which has the method

.build()

to return a driver.

builder: hamilton.driver.Driver = Builder().with_modules(...).with_adapters(...)
driver: hamilton.driver.Builder = builder().build()

The distinction might not be obvious from the examples because they are typically combined in one step

driver = (
   Builder()
   .with_modules(...)
   .with_adapters(...)
   .build()
)

Got it, thanks for the context!

FWIW, I feel like this interface (as above, or even just

dr = driver.Builder().with_modules(my_dataflow).build()

) is clunky/cumbersome. It makes doing one-off calcs in a notebook feel harder than it needs to be. I'd prefer to see an option like:

import hamilton
driver = hamilton.Driver(modules=[my_dataflow])
result = driver.execute(final_vars=['foo'])

or even even a helper for a one-off:

result = hamilton.execute(modules=[my_dataflow], final_vars=['foo'])

Sure, I can add (have added) simple wrappers for myself and my users. But I propose (for discussion) that simplifying the common-case is one way to lower the barrier to entry and increase adoption.

@Gavin Kistner It's definitely something valuable. We've discussed it a few times and the main issue is the backwards compatibility of

hamilton.driver.Driver.__init__

. The

Builder

was introduced to make sure we could expand the driver object (as in add more features) without breaking things each release Currently, it has

def __init__(
        self,
        config: Dict[str, Any],
        *modules: ModuleType,
        adapter: Optional[
            Union[lifecycle_base.LifecycleAdapter, List[lifecycle_base.LifecycleAdapter]]
        ] = None,
        allow_module_overrides: bool = False,
        _materializers: typing.Sequence[Union[ExtractorFactory, MaterializerFactory]] = None,
        _graph_executor: GraphExecutor = None,
        _use_legacy_adapter: bool = True,
    ):

Which means it must have a

config

arg THEN a list of

modules

and

modules=...

is not possible (early API mistakes are costly 😅) You need to call it

hamilton.driver.Driver({}, module_foo, module_bar)

## Potential solutions ### Change

Driver.__init__()

Can make a big breaking change with a more solid

Driver.__init__()

API. I think it's worth doing and we can have good error message for migration / upgrading. This would require a major release Hamilton 2.0 ### Add a new top-level entrypoint We discussed having something like

hamilton.execute()

. It's a good idea and definitely useful in quick scripts, notebooks, etc. Open questions: • I don't know if

.execute()

is the best method name if its signature differ from

Driver.execute()

• Should it have feature parity with

Driver

and

Builder

? (i.e., support adapters, materializers, config) Challenges: •

hamilton/__init__.py

imports no module because

hamilton/registry.py

needs to be imported first. This is to register plugins and more. To add

execute()

inside

hamilton/__init__.py

, we can't add module-level imports and we'll need to import things from within the function

Thanks! That's helpful context, parallel thinking, and highlighting some issues. I'mma noodle on this for a bit.

One of our ideas was to allow

hamilton.run()

to automatically capture the current frame / scope to build the dataflow. For example, this would automatically build the DAG defined in the module

def foo() -> int: ...

def bar(foo: int) -> float: ...

if __name__ == "__main__:
   import hamilton

   results = hamilton.run(["bar"])