This message was deleted.

Hi Jules, we don’t have any plan on that yet, how do you deploy BentoML api server today? have you tried using a DCGM-Exporter side-car?

Hey @Chaoyu we have 3 different Bento services deployed on a GKE cluster at the moment. We've set up actually set up the

DCGM

exporter (which was quite painful) but we're now trying to scale each service independently based on its own GPU consumption.

@Jules Belveze I assume the metrics you want to export are around GPU inference speed, usage of the models etc?

@Aaron Pham Yes exactly

We can definitely take advantage of

nvidia-smi

from the looks of things. I will get back to you as I find out more about this

Awesome thanks Aaron!