Hi everyone, how can I manage reading and writing...
# hamilton-help
h
Hi everyone, how can I manage reading and writing parts of my DAG from/to database. Let me give you the context here: I am building a DAG that has two main abstract functionalities: 1) training a model and updating this model, and I am saving the model's weights unstructuredly into a database, now I want to my DAG to read the model weights from the updates and update it using the incoming new data, how can I architect my DAG is a good way to accomplish that?
s
What's the challenge you're facing here?
Here's an example that shows modularity of a DAG with respect to training & inference -- not what you want; but if you wanted an idea around modularity this is one approach. Otherwise I think
@config.when
could be an option, or having an optional argument; or always checking for model weights, and if none are passed through starting fresh, and if some are passed through then you utilize them?
h
The challenge is that my DAQ should update a model once in a while. The model weights are saved in a database, so the DAQ should check if there's weights already saved in the database to be use while updating the model and then write the updated weights again to the database
if there's no data saved so that means the model hasn't been trained before and the DAG will save it for the first time
s
Copy code
def prior_model_weights(file_path: str) -> Weights | None:
   # check for weights -- load or return None

def trained_model(prior_model_weights: Weights | None, ...) -> Model:
   # train model -- check if weights is None or not ...

def write_model_file(trained_model: Model, file_path: str) -> dict:
  # write models
something like that shoudl work -- now if the
| None
annotation doesn't work, file a bug 🙂
👍 1