I’m deploying with Google Cloud and bentoctl and I’m using 32Gb and 8 CPU which I think is overkill considering the endpoints’ model objects aren’t that big (maybe 9 GB in total). I have the feeling (and hoping) I’m doing something naive like … storing to many copies of the model objects or something.

It’s hard to recreate with a smaller service. And I don’t want to share the code directly. Can I set up time with someone next week to do a video call? I would like to share my architecture to see if I’m doing something naive and to see if there’s an easy fix or if a deeper dive is needed.

I have a log statement in my

service.py

file right before I define my service

svc = bentoml.Service("proba-models", runners=list(runners.values()))

. When I go to serve with docker run, I proceed to see that log statement 96 times:

8 cpus x (10 runners + 1 "api_server" + 1 "cli")

. Does this mean that the service is getting created 96 times? Are each of these 96 instances getting copies of the runners?