Hi, is there a way to get all the graphs that are ...
# contributej
Jay
05/15/2024, 4:29 PMHi, is there a way to get all the graphs that are loaded into the driver?
e
Elijah Ben Izzy
05/15/2024, 4:30 PMHey! Mind clarifying a little more? We have a few tools to inspect/get data out. By graphs do you mean modules? Or the graph representations?
j
Jay
05/15/2024, 4:36 PMI wanted to get a list of data pipelines that are being represented in the current graph, example if I have two data pipelines
• Extraction from Source A
• Extraction from Source B
I wanted to see them both as a list.
e
Elijah Ben Izzy
05/15/2024, 4:38 PMHmm — interesting. By data pipelines do you mean the psecific sources? Or are they entirely independent?
j
Jay
05/15/2024, 4:39 PMthey are entirely independent graphs
e
Elijah Ben Izzy
05/15/2024, 4:39 PMOk, so this isn’t something that’s doable in one-step in Hamilton, but you can use graph processing to do it
Elijah Ben Izzy
05/15/2024, 4:42 PM Copy code
vars = dr.list_available_variables()
vars[0].dependencies
# list of dependencies Copy code
data_pipeline_1_start = "load_from_source_a"
data_pipeline_2_start = "load_from_source_b"dr.what_is_upstream_of(data_pipeline_1_start)👍 1
Elijah Ben Izzy
05/15/2024, 4:42 PMThis approach can help if you already know the independent sources. Otherwise you’ll want a simple graph partitioning algorithm (DFS) to break into “islands”
j
Jay
05/15/2024, 4:43 PMthanks, this sounds good 👍
e
Elijah Ben Izzy
05/15/2024, 4:44 PMYep! If you come up with some code you like that does this we could probably find a place in hamilton for these algorithms if you want to contribute them back 🙂
j
Jay
05/15/2024, 4:49 PMso the purpose I was trying this out for is basically create a CLI interface to execute data pipelines, I do know the pipelines coming in but if i can list them all that would be one place where we don’t need to make config edits and it can come directly from hamilton driver itself.
This might not be the intentional usage of Hamilton though
e
Elijah Ben Izzy
05/15/2024, 4:50 PMYeah, I think that’s very aligned. Its quite a reasonable way to model it. If you put it all in one big “dag”, then you can break it up and choose what to execute.
A thought: One way to think about this is to invert it — rather than thinking about “You have this DAG”, you can think “You have these ‘assets’“. You can then list it out (which will be the same as pre-defined pipelines), and choose which to execute.
+ @Thierry Jean who has worked on a CLI before.
j
Jay
05/15/2024, 4:50 PMI mean the UI (from my understanding) also does the same thing, so might not be the best way to go around this
e
Elijah Ben Izzy
05/15/2024, 4:51 PMYeah we highly encourage you to try out the UI 🙂 It’ll be a nice viz for this, allowing you to click/higlight the different DAGs
j
Jay
05/15/2024, 4:52 PMA thought: One way to think about this is to invert it — rather than thinking about "You have this DAG", you can think "You have these 'assets'". You can then list it out (which will be the same as pre-defined pipelines), and choose which to execute.e
Elijah Ben Izzy
05/15/2024, 4:53 PMAwesome! Yeah we’re starting to use the word “asset” to describe Hamilton/Hamilton pipelines. E.G. Hamilton focusees on the “Asset layer” — think about “assets” instead of “tasks” and your life gets easier 🙂
t
Thierry Jean
05/15/2024, 4:53 PMWill be out for PyCon this week, but happy to schedule a meeting and hear your thoughts about CLI execution. We have many pieces in place to ship that feature reasonably fast!
j
Jay
05/15/2024, 4:54 PMthanks @Thierry Jean all the best for the PyCon talk, lets talk about this when you are back