This message was deleted.

Were you able to verify each GPU has one instance of the model loaded?

Yes -

nvidia-smi

shows copies of each model on each GPU, and during load testing there is utilization across the GPUs. I'm getting much better throughput, but it's just not where we were hoping. Additionally, the throughput performance seems to be much more variable between load tests (i.e. when comparing the results of one load test to a subsequent one) than when using a one GPU instance.

I checked with the team and looks like the doc for working with multi-GPU is on our roadmap but still being worked on

@Chaoyu thanks for getting back to me! Good to know the docs are on the horizon - those will be helpful to have. We're using 1.1.6, and did use the

workers

to ensure that we got 1 copy of each runner on each GPU.

Hey sorry I missed this thread - just caught up

All good! Happy to hop on a call, but also happy to handle it async over Slack here. I think I'm mostly curious if it's on the horizon at all - we've got a bunch of models that take up significantly less than a single GPU's worth of memory and can load them all at the same time but I think there'd probably be at least a bit of gain from being able to use fractional GPU usage? That being said, my knowledge of exactly how GPU kernels work and how async calls to those kernels are is limited. So, I'm assuming it could also be the case that even if we could have fractional GPU usage and assign each model to a particular fraction of the GPU, it wouldn't actually make a difference because only one model will be able to call to the GPU kernels at once.