Hello, I'm new to Hamilton and I'm interested in adopting it to organize data science pipelines. I've dug around the docs for a few hours and made some cursory searches in slack, and I have some follow up questions now. Please forgive me if some of these have been answered elsewhere. One reason I'm interested in Hamilton is it seems to offer an intriguing way to organize processing/analytic steps across batch pipelines written by various data scientists and engineers. The passing of function names as nodes seems like a way to enforce a sort of schema across various processing jobs when sharing base components, as well as baked in documentation and knowledge management. I envision making an analytic store, where data scientists write their pipelines on a one module per pipeline basis, and a corresponding team analytic library, where commonly used nodes and subgraphs can be imported for various pipelines in the analytic store. Given consistent naming enforced on the function level, a setup like this should make it very clear what analytics are consuming what inputs, what sort of follow up processing on common nodes is being done, etc. to both reduce duplication of effort and to increase visibility of analytic deficiencies. I imagine writing a little code so a data scientist could query something like, 'what other pipelines use the

process_data_from_special_api

function, so they could find that a coworker's DAG brings in data from another source to enrich the data in their pipeline that would also be useful in their own, or that so many analytics use the same follow on steps that perhaps they should be modularized in a component library to increase consistency across analytics.· The thing is, this doesn't actually seem that easy to do in Hamilton, so I first have to ask, does the above setup I have in mind seem reasonable? From reading the documentation and doing a little experimenting, function reuse is tricky in general, or at least not intuitive. Imagine I made a module of reusable components that I import into

my_analytic

like

Builder().with_modules([component_library, my_analytic])

, but I really only want one function from

component_library

, say

process_special_data

. My understanding is, to hook it into the

my_analytic

dag, I would need to make sure I have upstream nodes that are the expected inputs to

process_special_data

and output nodes that will consume

process_special_data

as input. What happens to any extra nodes/subgraphs that I don't want from the module? Is there a way to just import the

process_special_data

function to make it explicit in the

my_analytic

calling code exactly what the analytic depends on? Is there any good way to further decorate an imported function like this, if I wanted to tag further metadata or change parameterizations specific to the analytic? If another analytic uses a function I want in mine, is there a better way to bring it into my own graph/model outside of copy pasting it? I have a separate, unrelated question about materialization and external APIs in general. Is materialization a concept that should only be used for initial and final I/O? If I grab data from a local parquet file using the built in parquet loader, process it, then grab data from an external API based on that processing to further enrich it, should I treat building a client, making the request, and processing the response as nodes in my graph, or should I write and register a data loader for the API that does this under the hood, so I can just decorate a node in my graph with

@load_from.my_api(request_params=source["parquet_processing"])

? Static materialization, if I understand correctly, is out of the picture for this case, since I don't know how to construct the request until I do the initial parquet processing? I don't really see any examples in Hamilton docs on data loading/materialization outside of fairly simple static examples. I appreciate any help on these questions.

One reason I’m interested in Hamilton is it seems to offer an intriguing way to organize processing/analytic steps across batch pipelines written by various data scientists and engineers. The passing of function names as nodes seems like a way to enforce a sort of schema across various processing jobs when sharing base components, as well as baked in documentation and knowledge management. I envision making an analytic store, where data scientists write their pipelines on a one module per pipeline basis, and a corresponding team analytic library, where commonly used nodes and subgraphs can be imported for various pipelines in the analytic store. Given consistent naming enforced on the function level, a setup like this should make it very clear what analytics are consuming what inputs, what sort of follow up processing on common nodes is being done, etc. to both reduce duplication of effort and to increase visibility of analytic deficiencies. I imagine writing a little code so a data scientist could query something like, ’what other pipelines use the

process_data_from_special_api

function, so they could find that a coworker’s DAG brings in data from another source to enrich the data in their pipeline that would also be useful in their own, or that so many analytics use the same follow on steps that perhaps they should be modularized in a component library to increase consistency across analytics.·

Yep that’s the theory!

Thank you for getting back to me so quickly! And thank you for that last link with the history. I need to ponder a little on what you said about logical vs agnostic assets and how I think it should fit with what I envision, and also look into

resolve

but I will try to follow up on some other points. > Can you elaborate more here? Is it something dynamic e.g. for each row? > or something else, e.g. needs to know file name at runtime (you can use

source(…)

here)? Sorry if I was imprecise. In the materializer examples in the docs, it seems like before you even create the graph, you need to know what parameters you pass the data loader, e.g.

path

. In my example, I wouldn't know how I wanted to call the API until I executed another node in the graph, e.g. hence I think using

source

in my example was correct? I think I see what you're saying now, you can still pass a

source

argument to the static materializer example to achieve this dynamic allocation anyway. > Not knowing exactly what you mean here. But assuming

process_special_data

is say some reusable function, e.g. like calculating

zero_mean

and you’d want to apply it to potentially multiple columns/inputs. Apologies again for imprecise language. My understanding is that

my_analytic.py

would use a builder like

Builder().with_modules([component_library, my_analytic])

, but even if I was just using one function out of a large

component_library

module, it wouldn't be immediately obvious which function I was using from viewing the code until I built the graph. It would be nice if I could explicitly select a few functions, like

Builder().with_modules_and_functions([component_library.function1, my_analytic])

. I think I'm asking for something like https://github.com/DAGWorks-Inc/hamilton/issues/1268, so perhaps this is already in the works? I think I would also like to see this in the case of, what if I want to pull a few functions from another analytic

my_analytic_2

on an ad hoc basis? Refactor shared components is probably the first logical answer, but perhaps the core component library has separate permissions from the analytic store library. I'm worried that building with

my_analytic_2

and

my_analytic

modules wholesale, there might be a lot of collisions and spurious sub-dags and nodes being created. I think doing this can help speed development of analytics and pipelines and increase sharing, as it would be easy to 'test drive' nodes that another data scientist has created

> My understanding is that

my_analytic.py

would use a builder like

Builder().with_modules([component_library, my_analytic])

, but even if I was just using one function out of a large

component_library

module, it wouldn’t be immediately obvious which function I was using from viewing the code until I built the graph Yep - what’s your concern here though? It should be clear given the naming of what’s required to be run though? Regarding 1268, would you mind adding your context / concerns to that issue please? Your request doesn’t sound unreasonable to add. You could start to see the potential for sets of functions to be grouped in a module and be referenced like that in a builder too… > I’m worried that building with

my_analytic_2

and

my_analytic

modules wholesale, there might be a lot of collisions and spurious sub-dags and nodes being created. . I think doing this can help speed development of analytics and pipelines and increase sharing, as it would be easy to ‘test drive’ nodes that another data scientist has created. Spurious parts of the graph should have no impact, other than visualization — thought there is the visualize execution function on the driver that would weed out anything not on the path (there’s also the Hamilton UI which could get more features here to help). Collisions could happen yes, though there is a feature to allow taking the “last defined” function if it’s in multiple modules. Otherwise I can see where you’re going. Would have to think if this would lead to any anti-patterns (your driver might take in a real piece meal of things… but maybe that’s okay). As a side note, you might also like the 2nd half of this talk I did on .

Thanks you so much for patiently answering my questions. I'll go ahead and comment on the github issue and continue to experiment with hamilton!