This message was deleted.

Wow, nice idea!

@Bo would this be of interest?

@Nicolas Jaccard Good suggestions. Could you tell me more about the specific use-cases for model caching/reloading? "hot" load model and caching are on our roadmap for yatai, when yatai can take on more responsibilities.

The use-case if for the serving of multiple models on a single GPU node. Caching / reloading allows one to serve more models than what would be possible if all models had to reside in GPU RAM simultaneously.