This message was deleted.
# ask-for-helps
Slackbot
01/26/2024, 4:01 PMThis message was deleted.
o
Oliver Haney
01/26/2024, 4:12 PMDecided to use my head and search this slack for answers. I have some paths to explore now, looks like lazy loading models is supported out of the box. Thanks!
j
Jian Shen Yap
01/26/2024, 4:34 PMWhat model do you want to use? @Aaron Pham maybe you could help with some suggestions too
o
Oliver Haney
01/26/2024, 4:37 PMI’m starting with
databricks/dolly-v2-3bOliver Haney
01/26/2024, 4:37 PMI’m open to using other models though, I’m not sure which one will be best for my use case yet
Oliver Haney
01/26/2024, 4:42 PMTo explain a little more, I want to deploy my bento container to a gpu accelerated kubernetes node group, and load the model at runtime via and EFS volume. This way I can reduce the image size
c
Chaoyu
01/26/2024, 11:13 PMOne option is to configure the node with dependencies and run openllm start command directly, without packing all dependencies and model files into the image.
Fast scaling of large container images on GPU cluster needs a stack of complicated optimizations and that’s one of the reasons we built BentoCloud :)