This message was deleted.

are you using the latest version of bentoml 1.0.8?

cc @sauyon

Thanks for reporting @Sudeep Ghimire. We are investigating this recent surge of

Broken pipe

issues. Will update once we have identified a fix.

Above exception was with bentoml 1.0.5 and have just upgraded to bentoml 1.0.8 and got some failures with new exception `raise RemoteException(f"Failed to connect to runner server.") from e
bentoml.exceptions.RemoteException: Failed to connect to runner server.

Details trace is:

2022-11-02T13:23:31+0000 [ERROR] [api_server:2] Exception in ASGI application
Traceback (most recent call last):
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/uvicorn/protocols/http/h11_impl.py", line 405, in run_asgi
    self.scope, self.receive, self.send
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/uvicorn/middleware/proxy_headers.py", line 78, in __call__
    return await <http://self.app|self.app>(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 86, in __call__
    raise exc from None
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/uvicorn/middleware/message_logger.py", line 82, in __call__
    await <http://self.app|self.app>(scope, inner_receive, inner_send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await <http://self.app|self.app>(scope, receive, _send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/bentoml/_internal/server/http/instruments.py", line 176, in __call__
    await <http://self.app|self.app>(scope, receive, wrapped_send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/opentelemetry/instrumentation/asgi/__init__.py", line 487, in __call__
    await <http://self.app|self.app>(scope, otel_receive, otel_send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/bentoml/_internal/server/http/access.py", line 126, in __call__
    await <http://self.app|self.app>(scope, receive, wrapped_send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
    raise exc
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
    await <http://self.app|self.app>(scope, receive, sender)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/routing.py", line 680, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/routing.py", line 427, in handle
    await <http://self.app|self.app>(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/fastapi/applications.py", line 270, in __call__
    await super().__call__(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/applications.py", line 124, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/errors.py", line 184, in __call__
    raise exc
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/errors.py", line 162, in __call__
    await <http://self.app|self.app>(scope, receive, _send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 75, in __call__
    raise exc
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/middleware/exceptions.py", line 64, in __call__
    await <http://self.app|self.app>(scope, receive, sender)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 21, in __call__
    raise e
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/fastapi/middleware/asyncexitstack.py", line 18, in __call__
    await <http://self.app|self.app>(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/routing.py", line 680, in __call__
    await route.handle(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/routing.py", line 275, in handle
    await <http://self.app|self.app>(scope, receive, send)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/starlette/routing.py", line 65, in app
    response = await func(request)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/fastapi/routing.py", line 232, in app
    dependant=dependant, values=values, is_coroutine=is_coroutine
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/fastapi/routing.py", line 160, in run_endpoint_function
    return await dependant.call(**values)
  File "/home/ubuntu/VideaCode/BentoML-Evaluation/app/routes/infer_routes.py", line 26, in predict
    demo_rbl_runner.infer.async_run(pil_image=pil_img)
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/bentoml/_internal/runner/runner.py", line 56, in async_run
    **kwargs,
  File "/opt/conda/envs/inference-server/lib/python3.7/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 195, in async_run_method
    raise RemoteException(f"Failed to connect to runner server.") from e
bentoml.exceptions.RemoteException: Failed to connect to runner server.

Please also count me as someone that is experiencing “broken pipe” issues. ➕ Happy to assist in any way I can.

We're currently working on getting a fix for this, it's definitely our highest priority issue right now.

We found out that both

aiohttp

and

httpx

clients raise this error under high concurrency. The problem goes away if we switch to using the sync API of the

aiohttp

and

httpx

client. Using sync API is not ideal because it impacts throughput and disallows making concurrent calls to runners. Given the recency of the issue, we have also tried reverting relevant libraries like

aiohttp

,

startlette

, an

uvicorn

but that didn’t seem to help. We will keep the community updated with our investigations.

Oh that’s a bummer :( hope some resolution will come up

@Mike Kuhlen @Sudeep Ghimire We identified and fixed this issue in

v1.0.9

. Please see if it solves this problem on your end. Thanks for your patience. https://github.com/bentoml/BentoML/releases/tag/v1.0.9

That’s awesome.. thanks for the update and the fixes. Will run experiment and let you know how it goes.

I will check it out too!

Quick update. Please use

v1.0.10

instead. We did a quick patch to

v1.0.9

for an unrelated bug fix. https://github.com/bentoml/BentoML/releases/tag/v1.0.10