https://bentoml.com logo
Join Slack
Powered by
Hi.. I have this curious issue where BentoML rever...
# ask-for-help
a
Hi.. I have this curious issue where BentoML reverts timeout to 60 seconds after the first API call: 2025-04-07T145542-0400 [INFO] [entry_serviceSentenceTransformers1] 127.0.0.1:51128 (scheme=http,method=POST,path=/encode,type=application/json,length=201092) (status=200,type=application/json,length=3425048) 143723.858ms (trace=ee7d3af3e7d7d5e92254467826ea6f50,span=38faf12024132b83,sampled=0,service.name=SentenceTransformers) ..... <-------------- After this call all of them fail bentoml.exceptions.ServiceUnavailable: process is overloaded 2025-04-07T145706-0400 [INFO] [entry_serviceSentenceTransformers1] 127.0.0.1:51302 (scheme=http,method=POST,path=/encode,type=application/json,length=201092) (status=503,type=application/json,length=74) 60017.688ms (trace=4f3ac9e760c18b55a3be7000bacf5600,span=2413b6e7cb2c4293,sampled=0,service.name=SentenceTransformers) ..... bentoml.exceptions.ServiceUnavailable: process is overloaded 2025-04-07T150155-0400 [INFO] [entry_serviceSentenceTransformers1] 127.0.0.1:51596 (scheme=http,method=POST,path=/encode,type=application/json,length=201092) (status=503,type=application/json,length=74) 60005.034ms (trace=22be9ba69e6af428f8e7586232f5390c,span=4749414bffed68e4,sampled=0,service.name=SentenceTransformers)
j
Hello, could you share your service code?
a
Sure one sec. It's a vanilla sentence transformer using model from hugging face
Untitled.py
Untitled.py
f
it hits the 
max_latency_ms
for batchable endpoints, not the 
timeout
limit, the latter only returns a 504 error
raise the value in 
@bentoml.api(max_latency_ms=...)
a
Hi Frost, thank you. Raised the value as follows, same behavior.
Copy code
@bentoml.service(
    timeout=300,
    traffic={"timeout": 300},
    max_latency_ms=300_000
)
class SentenceTransformers:
I.e. first request succeeds, second request fails at 60 second mark
Copy code
2025-04-08T08:40:24-0400 [INFO] [entry_service:SentenceTransformers:1] 127.0.0.1:64446 (scheme=http,method=POST,path=/encode,type=application/json,length=201092) (status=200,type=application/json,length=3425048) 151789.005ms (trace=5491b6186c17e30718f05b13090ed97c,span=18943e3dc8b2c9f2,sampled=0,service.name=SentenceTransformers)
2025-04-08T08:41:36-0400 [ERROR] [entry_service:SentenceTransformers:1] Exception on /encode [POST] (trace=312639eb08d8617a4ddb3e6b132f99ff,span=63493c48e2a8312c,sampled=0,service.name=SentenceTransformers)
Traceback (most recent call last):
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/bentoml/_internal/marshal/dispatcher.py", line 189, in _func
    r = await self.inbound_call(data)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/bentoml/_internal/marshal/dispatcher.py", line 334, in inbound_call
    return await future
           ^^^^^^^^^^^^
asyncio.exceptions.CancelledError

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/_bentoml_impl/server/app.py", line 604, in api_endpoint_wrapper
    resp = await self.api_endpoint(name, request)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/_bentoml_impl/server/app.py", line 688, in api_endpoint
    output = await self.batch_infer(name, call_args, call_kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/_bentoml_impl/server/app.py", line 596, in batch_infer
    return await self.dispatchers[name](
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/bentoml/_internal/marshal/dispatcher.py", line 191, in _func
    return self.fallback()
           ^^^^^^^^^^^^^^^
  File "/Users/guseant/.pyenv/versions/3.12.6/lib/python3.12/site-packages/_bentoml_impl/server/app.py", line 118, in fallback
    raise ServiceUnavailable("process is overloaded")
bentoml.exceptions.ServiceUnavailable: process is overloaded
2025-04-08T08:41:36-0400 [INFO] [entry_service:SentenceTransformers:1] 127.0.0.1:64590 (scheme=http,method=POST,path=/encode,type=application/json,length=201092) (status=503,type=application/json,length=74) 60015.561ms (trace=312639eb08d8617a4ddb3e6b132f99ff,span=63493c48e2a8312c,sampled=0,service.name=SentenceTransformers)
oh wait, it's an API decorator....
Yep, that worked.
Thank you @Frost Ming
2 Views
Open in Slack
PreviousNext
Powered by