Hello, I am using BentoML version v1.1.11. I configured my BentoML service to use one worker for the api_server and it appears that this worker is using the GPU during a request. Is it possible to configure it to not use GPU?

It may have to do with accidentally importing PyTorch, could you share your service definition code?

I am not able to share my code, but it appears that PyTorch is not being imported. Is it not normal for the worker to use GPU? It appears to be using the GPU during a few lines in a model inference endpoint.

really depends on how you structure your code, you can definitely choose to use GPU in API server workers but usually the pattern is to use GPU for model inference in a standalone model worker (runner)

Yes, the goal is for only my model runner to use GPU and everything else to not use GPU. Currently, both my model runner and what appears to be the BentoML service API worker are using GPU and I am unsure how I can configure the worker to not use GPU.

Got it, this is more about user code, not BentoML configuration. BentoML don’t prevent user api server code from accessing GPU (unless using distributed deployment on BentoCloud where you can assign different resource shape for api workers and runners)

I see. So would I have to manually figure out a way to prevent the worker from using GPU? My current approach is to set an environment variable to control when GPUs can be available or not.

That works too, although you should have control directly in the service code, right?

Yes, I meant that I would change the environment variable within the service code.

Yes, depends on the ML framework you’re using, but usually they respect the env var. be careful if the framework checks the env var during library import or model loading

So to recap, BentoML's api_server worker cannot be configured to not use GPU, so a manual method needs to be done, correct?

That’s correct, this may imply there’re other issues with the service code - do you want to load the model and run on CPU in the api worker?

I would like the process for the model runner to be the only process utilizing the GPU.

API server serves user defined Python code, whether or not it uses GPU only depends on user code

Ah, I see. So if my API worker is using GPU right now, it would be something related to the user code I wrote and not as much BentoML's built-in configurations or features? And then in that case I should look into either manually controlling GPU access or seeing if some part of the code I wrote is causing the worker to use GPU? The model runner seems to be correctly using the GPU, it's just that the API worker is as well and that is not desired.

Yes that’s correct