Zhenya Kovalyov
09/04/2025, 12:57 PM@save_to.parquet(...) sprinkled around my pipelines, and in order to trigger the saving routine, i need to explicitly call out save.<name> in final_vars in driver.execute
question - is there a way i can modify my driver.execute call, so that all vars that are decorated with save_to get persisted?
thank you very much in advance!Thierry Jean
09/04/2025, 1:10 PMdef _include_saver_nodes(dr: Driver, requested_nodes: list[str]) -> list[str]
selected_saved_nodes = []
for node in dr.list_available_nodes():
# save nodes will have the prefix `save.`
if "save." not in node.name:
continue
_, _, base_node_name = node.name.partition(".")
if base_node_name in requested_nodes:
selected_saved_nodes.append(node.name)
return requested_nodes + selected_saved_nodes
dr = Builder().with_modules(...).build()
requested_nodes = _include_saver_nodes(dr, ["customers", "sales_per_month"])
dr.execute(requested_nodes)Zhenya Kovalyov
09/04/2025, 1:22 PMElijah Ben Izzy
09/04/2025, 5:15 PM