This message was deleted.

My setup: • 4 models in the bentoml service ◦ 4 instances of model 1 ◦ 4 instances of model 2 ◦ 2 instances of model 3 ◦ 2 instances of model 4 ◦ all on GPUs • starting service with 20 api workers • hitting service with ~20-30 parallel workers Getting service busy error ~80% for most of the requests.

Is this the first request that you got?

you mean error messag?

yes

This is the full traceback:

2022-11-22T20:06:26+0000 [ERROR] [api_server:2] Exception on /generate-phrases-generic [POST] (trace=a4728477755439c7a2f2756980de6d9b,span=6228acb99fadafd4,sampled=0)
Traceback (most recent call last):
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/server/http_app.py", line 324, in api_func
    output = await api.func(input_data)
  File "/disk1/projects/tzhang/cainet/bentoml/bentos/cainet_model_server_prod/4ltywcdkowekscga/src/cainet_model_server_api.py", line 338, in generate_phrases_generic
    outputs = await generic_translation_runner.generate_phrases.async_run(request_data)
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/runner/runner.py", line 53, in async_run
    return await self.runner._runner_handle.async_run_method(  # type: ignore
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 207, in async_run_method
    raise ServiceUnavailable(body.decode()) from None
bentoml.exceptions.ServiceUnavailable: Service Busy
2022-11-22 20:06:26,383 - bentoml._internal.server.http_app - ERROR - Exception on /generate-phrases-generic [POST]
Traceback (most recent call last):
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/server/http_app.py", line 324, in api_func
    output = await api.func(input_data)
  File "/disk1/projects/tzhang/cainet/bentoml/bentos/cainet_model_server_prod/4ltywcdkowekscga/src/cainet_model_server_api.py", line 338, in generate_phrases_generic
    outputs = await generic_translation_runner.generate_phrases.async_run(request_data)
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/runner/runner.py", line 53, in async_run
    return await self.runner._runner_handle.async_run_method(  # type: ignore
  File "/disk1/projects/tzhang/cainet/venv/lib/python3.8/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 207, in async_run_method
    raise ServiceUnavailable(body.decode()) from None
bentoml.exceptions.ServiceUnavailable: Service Busy

when did you receive this err log?

it’s during when I’m hitting the server with ~20 parallel workers for load testing.

same problem, @Yilun Zhang did you find solution?

I was having this issue even with only 20 users or 10 users.

I believe service unavailable means that the runner server doesn't think that it can serve your request within the max latency you set, and is therefore cancelling the request---does raising the max latency help?

Yeah, that’s exactly what I was thinking but i felt like with the current setup it should be raising this issue. I’m getting this even with ~10 workers. Raised latency to 10000 and still the same, but it’s not acceptable for me because the latency is too large. i also tried to host 8 models of each but still no luck with this.

The same as in you're still getting service unavailable errors?

yes, so I’m wondering if bentoml team has any baseline for # of api workers and number of models to host for certain RPS and model response time? For example, if I have a text-generation model with single request response time of ~250ms, and if i want to achieve ~30-40 RPS with ~max response time of ~500ms, then what could be the ideal setup?

I think unfortunately batched request time varies depending on the kind of model, so it's hard to say. @larme or @Aaron Pham might have more context, though. Ballpark I'd note that to achieve 30-40rps with that latency naively (without batching) I'd expect you to need ~15-20x the current resources to match that throughput.

I’m doing batching, so that’s probably the main latency (waiting to get requests to get batched). I tried to scale the resource a bit more by hosting ~20 models of each (total 4 models). Not getting service busy anymore but having really bad response time. I guess it’s moving towards better direction. I will experiment more. The mystery was, before when using 0.13, I was able to achieve good rps with only 3 GPUs and 6 workers in total, so this was my main concern and question.

Hm, interesting. I think @Jiang might know more about the 0.13 behavior but that's definitely concerning.

How large is your text generation model?

It's the Pegasus model (2.2Gb)