BentoML

Hey <@U0476TQKUJC> , how are you serving BentoML? are you running it in a docker or just bentoml serve?

i think its some configuration mismatch thats causing intermittent issues.. we are also noticing come intermittent `Server Busy` , which seems to lead to indicate some configuration that can help us to better tune the performance

The 503 Server Busy error message is something that we are working to improve. It's usually due to our dispatcher cancelling request if the runner is overloaded. 

For your case of the disconnected error, do you mind creating a minimal reproducible example for us?

&gt; The 503 Server Busy error message is something that we are working to improve. It’s usually due to our dispatcher cancelling request if the runner is overloaded.
Really looking forward for this improvement.

&gt; For your case of the disconnected error, do you mind creating a minimal reproducible example for us?
Recreating might be difficult since it has been very intermittent on our end, specifically as load increases. But will spend some time on our end to see of we can make a reproducible example

could the `aiohttp.client_exceptions.ServerDisconnectedError: Server disconnected` be related to `aiohttp` itself? Will adding a sleep in between requests resolve this issue? ref: <https://github.com/aio-libs/aiohttp/issues/4549>

&gt; Recreating might be difficult since it has been very intermittent on our end, specifically as load increases.
maybe you could create the simplest bento and load test it, and see if the same error is reproduced?

<@U03UY1JB269> there is an issue reported in bentoml repo <https://github.com/bentoml/BentoML/issues/3669> we had seen this (and seems some else too) during load test on our end.