This message was deleted.
# ask-for-helps
Slackbot
09/07/2023, 3:34 PMThis message was deleted.
c
Chaoyu
09/07/2023, 3:50 PMAPI server workers and runner workers are controlled differently in BentoML
Chaoyu
09/07/2023, 3:50 PMChaoyu
09/07/2023, 3:51 PMWhat does your Runner do? Most of the ML frameworks/libraries can actually take advantage of multiple CPU cores
n
Nicolas Jaccard
09/07/2023, 3:56 PMIt's running inference but we have a limited memory budget per pod in Kubernetes and having 5 workers makes us hit that limit pretty frequently
Nicolas Jaccard
09/07/2023, 3:57 PMEven though we can take advantage of the multithreading I would rather limit that number if possible
Nicolas Jaccard
09/07/2023, 3:59 PMIf I understand correctly the documentation you linked,
cpuNicolas Jaccard
09/07/2023, 4:02 PM-e BENTOML_CONFIG_OPTIONS='runners.resources.cpu=1'Nicolas Jaccard
09/07/2023, 4:12 PMIs that what you had in mind @Chaoyu? (thanks for your help with this, much appreciated ๐ )
j
Jian Shen Yap
09/07/2023, 4:23 PMare you seeing 5 workers for runners or the server ?
n
Nicolas Jaccard
09/07/2023, 4:28 PMNow that I looked closer at the logs, it appears to be multiple api servers?
Copy code
2023-09-07T16:26:59+0000 [INFO] [api_server:2] Service loaded from Bento directory: bentoml.Service(tag="pipeline-dr-regression-icdr:xd5lqpcnsk5h47qe", path="/home/bentoml/bento/")
2023-09-07T16:27:00+0000 [INFO] [api_server:4] Service loaded from Bento directory: bentoml.Service(tag="pipeline-dr-regression-icdr:xd5lqpcnsk5h47qe", path="/home/bentoml/bento/")
2023-09-07T16:27:00+0000 [INFO] [api_server:5] Service loaded from Bento directory: bentoml.Service(tag="pipeline-dr-regression-icdr:xd5lqpcnsk5h47qe", path="/home/bentoml/bento/")
2023-09-07T16:27:00+0000 [INFO] [api_server:1] Service loaded from Bento directory: bentoml.Service(tag="pipeline-dr-regression-icdr:xd5lqpcnsk5h47qe", path="/home/bentoml/bento/")
2023-09-07T16:27:00+0000 [INFO] [api_server:3] Service loaded from Bento directory: bentoml.Service(tag="pipeline-dr-regression-icdr:xd5lqpcnsk5h47qe", path="/home/bentoml/bento/")Nicolas Jaccard
09/07/2023, 4:29 PMThough I am setting the number of workers to 1 so not sure why it would spawn 5 of them?
j
Jian Shen Yap
09/07/2023, 4:48 PMhmm. this should work though.-e BENTOML_CONFIG_OPTIONS='api_server.workers=1'
Jian Shen Yap
09/07/2023, 4:48 PMheres a local testing
Jian Shen Yap
09/07/2023, 4:49 PM Copy code
#############################
# ๐ฟ๐ฟ๐ฟ๐ฟ๐ฟ๐ฟ๐ฟ๐ฟ๐ฟ Simple Service
#############################
import bentoml
svc = bentoml.Service("simple")
@svc.api(input=bentoml.io.Text(), output=bentoml.io.Text())
async def count(input_text:str) -> str:
return "yo"n
Nicolas Jaccard
09/07/2023, 5:09 PMLet me give it a shot
Nicolas Jaccard
09/07/2023, 6:36 PMThis seems to work on my local machine, will now test on Kubernetes
Nicolas Jaccard
09/07/2023, 6:38 PMQuick question - is the default to have N api server workers and M runner processes? It seems like if I only set one of them I still get multiple versions running concurrently
j
Jian Shen Yap
09/07/2023, 9:30 PMThe default is api server process = number of cores. as for runner, it depends on the runner configuration. For
For runner, you can see here
https://docs.bentoml.com/en/latest/guides/scheduling.html
๐ 1
n
Nicolas Jaccard
09/08/2023, 11:25 AMThis worked, thank you! ๐
3 Views