This message was deleted.

Hey! So yes, fully supported. There are a lot of different ways to do this, and it depends on what you’re trying to do. 1. If its a matter of some functions only existing in certain specific DAGs, you can use

config.when

: https://hamilton.readthedocs.io/en/latest/reference/api-reference/available-decorators.html#config-when. For instance…

@config.when(region='UK')
def function__us(dep_1: ..., dep_2: ...) -> ...:
    """DAG only gets constructed with this version when the region is set to US"""

@config.when_not(region='UK')
def function(dep_3: ...) -> ...:
    """DAG gets constructed with this version when the region is not US (default)"""

Then in the driver, you just pass config in as the first argument:

dr = driver.Driver({"region" : "US"})

There are some more advanced features with

config

if you want — you can actually use it to dynamically resolve any decorator using

resolve

(https://hamilton.readthedocs.io/en/latest/reference/api-reference/decorators.html#resolve), but I’d stay away from it if you really need it. One other thing to note is that

config

is different from

inputs

—

config

is used to build the shape of the DAG, and

inputs

is used to pass data in (although anything in

config

is also passed into

inputs

). Does this get at what you’re trying to do?

If you’re parameterizing functions with different values. You could also model those values in code as functions themselves, and curate them into different modules. Then at driver time, pass in the right module to make the DAG, e.g. one module representing config, then other modules representing the common library of transforms.

I am more interested in changing the parameter values based on config. I have already parameterized all the hamilton functions. What I am trying to do is change the values passed to those parameters. I think what @Stefan Krawczyk is suggesting, is more inclined to what I thought. But I still don’t have clarity how to achieve it. If you can provide some dummy logic to help me out here, would be really helpful.🙂

OK, to clarify — you have a DAG that’s fixed, but you want to have a different set of parameters to be passed in, and you don’t want the user to have to remember them, right? I think what @Stefan Krawczyk said makes sense. You can imagine:

#common.py
def some_logic(external_input: ...) -> ...:
    ...

#setting_1.py
def external_input() -> ...:
   return SOME_VALUE

#setting_2.py
def external_input() -> ...:
   return SOME_OTHER_VALUE

Then in the driver:

immport common, setting_1, setting_2
dr = driver.Driver({}, common, setting_1)
dr_2 = driver.Driver({}, common, setting_2)

This is pretty simple, but you can imagine having some pretty complex configurations put in..

Ok, let say I have below function defined.

#common.py
@parameterize(
  datasetA={...},
  datasetB={...},
)
def make_dataset(external_input: ...) -> ...:
    ...

Now my DAG has both datasetA and datasetB nodes in all cases, but

external_input

for datasetA and datasetB will differ from case to case. Can this be configured? I don’t want to split the

make_dataset

into two hamilton functions`datasetA` and

datasetB

.

Oh, got it. So, yeah, I think this is pretty easy if I understand correctly:

@parameterize(
    datasetA={"external_input" : source("external_input_dataset_a")},
    datasetB={"external_input" : source("external_input_dataset_b")},
)
...

Then, all you need is to pass in

external_input_dataset_a

and

external_input_dataset_b

to the driver:

dr = driver.Driver(common)
dr.execute(
    ["datasetA", "datasetB"], 
    inputs={"external_input_dataset_a" : ..., "external_input_dataset_b" : ...})

or you can define them in the other modules…

Ok, now I got it. Thanks for the help!👍

Yeah! Looks like the solution is way simpler than anything we’ve been suggesting 😆

Yeah, multiple ways to do the same thing. This one is much cleaner i think.

Agreed — it limits the surface area to specific, enforcable (through the framework) contracts

I would be defining config in separate modules, instead of passing as inputs, bcoz I have to configure almost all nodes in the DAG😅

Yep! DAG is meant to be configurable — up to you to decide how to expose it to your end users

yep, the ability to determine what constitute inputs/edges of the DAG by including or excluding a module I think is one of the properties I like most about Hamilton.