This message was deleted.
# ask-for-helps
Slackbot
08/30/2023, 11:46 PMThis message was deleted.
c
Chaoyu
08/30/2023, 11:48 PMthe default API worker count probably didn’t make sense in your case with a host that have big number of cores
Chaoyu
08/30/2023, 11:49 PMlazily import modules in the service definition file can help as well - try avoid importing large libraries like PyTorch in
service.pym
Mykyta Holubakha
08/30/2023, 11:50 PMI only import
Copy code
import bentoml
from <http://bentoml.io|bentoml.io> import Multipart, Image as BImage, JSON
from PIL import Imagec
Chaoyu
08/30/2023, 11:51 PMAPI workers shouldn’t be taking GPU memory then 🤔
Chaoyu
08/30/2023, 11:51 PMhappy to help take a look at your service definition file, feel free to DM me the code if it’s sensitive
m
Mykyta Holubakha
08/30/2023, 11:51 PMMaybe bentoml.transformers.get causes implicit imports
👍 1
Mykyta Holubakha
08/30/2023, 11:51 PMhttps://paste.sr.ht/~hummer12007/55d9fbcc2268d6004c0e1b01b9cf3449de588585 (changed save_model to load a quantized 8-bit version since)
Mykyta Holubakha
08/30/2023, 11:51 PMThanks a lot!
c
Chaoyu
08/31/2023, 12:00 AMyes you’re right,
bentoml.transformers.gettransformersChaoyu
08/31/2023, 12:03 AMI will discuss this more with the team and benchmarks the cost of importing this type of libraries in api workers. For now, I’d suggest limit the number of api workers using either the configuration file or the CLI argument
m
Mykyta Holubakha
08/31/2023, 12:13 AMThanks!
3 Views