Hey guys, one more question regarding local development. When I start local server with

bento serve

and my bento configuration uses more than one model (and each model requesting GPU resources obviously), does it mean i need more than one GPU or they can load into a single one if there is enough VRAM? I'm getting

No CUDA GPUs are available

errors While this runs ok:

>>> import torch
>>> torch.cuda.is_available()
True
>>> torch._C._cuda_init()

and

nvidia-smi

reports drivers and GPU status is ok Thanks!

try run the code without bentoml and see what happens if you have mac you can also try load it to mps ( apple's GPU) and debug locally

It doesn't let me