Hi there! I'm wondering if there is any best practice guidance about dynamically generating DAGs. I am trying to implement hamilton into a statistics engine that I have, the following steps are undertaken: 1. User defines a configuration file with a list of metrics (yaml file) 2. User calls the engine with this config file, creating a collection of metrics to be run. 3. Engine runs through these metrics, returns a bunch of results to the end user. Hamilton would solve some pain points that I have around parallel runs, caching and visualization of runs, however I'm finding it a bit difficult to find any examples of this on the GitHub (I might just not be able to read...). I have been using ModuleType and setattr to dynamically generate the module but this feels a bit contrived to me. Thanks for any help! Below is an example DAG that I would like to generate depending on the config file inputs: load_data -> preprocess_data -> filter_data -> run_metric -> return_result Currently each step has different configuration available and is not required for each metric, therefore a dynamic approach is best.

@Jonas Meyer-Ohle thanks for the Q. Could you help give an example (pseudo code is fine) of the config file contents, and then the code you want it to create and then run to better understand please?

Of course! The following is a sample dataset:

customer_id,segment,age,income,balance,transactions,is_active
1,high_value,56,82943.05863806607,1685.9201673377806,83,True
2,high_value,27,35977.22356123622,1903.5720344516496,61,True
3,high_value,59,49342.221356986505,1550.519270467577,58,True
4,high_value,46,23271.912403430415,2668.6989342760176,66,True
5,high_value,78,40096.08135406822,1789.1313721499523,11,True
6,high_value,36,64040.777196341194,2849.7783030529235,43,True
7,high_value,34,69644.799139234,2796.777957856947,87,True
8,high_value,69,67209.85823731136,3906.28427701675,1,True

The following is an example config file:

metrics:
  metric_1:
    config:
      metric_name: mean
      name: mean_income
      segment: [segment]
      variable: income
      description: Mean income by customer segment
      dataset: "example_customer_data.csv"
  
  metric_2:
    config:
      metric_name: mean
      name: mean_age
      variable: age
      description: Mean customer age across all segments
      dataset: "example_customer_data.csv"

And how I would want the module to be generated:

import polars as pl
from hamilton.function_modifiers import parameterize, source, value, parameterize_values
from polars.lazyframe.group_by import LazyGroupBy

REQUIRED_COLUMNS = {
    "metric_1_required_columns": ["income", "segment"],
    "metric_2_required_columns": ["age"],
}


@parameterize_values(parameter="column_names", assigned_output=REQUIRED_COLUMNS)
def required_columns(input_lf: pl.LazyFrame, column_names: list[str]) -> pl.LazyFrame:
    return input_lf.select(column_names)


@parameterize(
    group_by_metric_1={"input_lf": source("metric_1_required_columns"), "group_by_col": value("segment")},
    group_by_metric_2={"input_lf": source("metric_2_required_columns"), "group_by_col": value("none")},
)
def group_by(input_lf: pl.LazyFrame, group_by_col: str) -> LazyGroupBy:
    return input_lf.group_by(group_by_col)

# Continue with metric calculation parameterization and result collection

So for any metric that uses the same dataset, I would generate a parameterized function that selects the required columns, then creates a group by which is then used by the actual metric implementation to generate a result. In this case that's just a mean value, but the actual engine has more complicated metrics such as adfuller etc with RAG styling. Data would be loaded and saved using materializers.

So, Hamilton is generally designed to be "code-first", but there are plenty of ways to do this! Specifically, there are three broad approaches here. I'll give the high-level and you can look through the trade-offs. LMK which appeal and I can write some pseudocode! Otherwise will write out something for the one I would prefer in your case: 1. Literal code generation -- E.G. templatize this -> your code above then run it. Actually not crazy, and kind of cool. But a bit of a project... Note that you could actually generate a single function per, and they could all out to 2.

@resolve

-based approaches a.

@resolve

allows you to take a configuration item an use it to shape the graph b. E.g. produce a decorator from config c. In this case you could either: i. (try this first, see next item) -- Produce the code you have above, with the

@parameterize

and

@parameterize_values

wrapped in resolve that would output the decorator with n arguments (one for each metric in groupby/required_columns) ii. Use a subdag/parameterized_subdag decorator to build out metric subdags 3. Runtime-dynamism -- use

Parallelizable

Cheers that really helped! I'll give #2 a try, but might have to switch to code generation for some of the more complex parts. Had a look at @resolve before asking but your example really helped me out 🙂

Awesome! Surprised, I was writing code in the late evening on slack, it never works when I do that 🤣 Will try to get you an RC version tonight to test it out.

Cheers!

@Jonas Meyer-Ohle @Elijah Ben Izzy should be able to get something out this week; to confirm do you need new docker containers? or just the

sf-hamilton-ui

library? or both?

Anything is fine for me, I can build the container if needed, thanks!

OK, released

sf-hamilton-ui==0.0.17

— sorry for the delay!

@Jonas Meyer-Ohle docker images should also have been pushed