This message was deleted.
# ask-for-helps
Slackbot
12/06/2022, 9:33 PMThis message was deleted.
s
sauyon
12/06/2022, 9:35 PMCan you provide us with your server logs?
l
Liwen Huang
12/06/2022, 9:37 PMdo you mean from http://localhost:3000/metrics?
s
sauyon
12/06/2022, 9:37 PMNo, the output that's printed from the
docker runl
Liwen Huang
12/06/2022, 9:41 PMbentoml.exceptions.RemoteException: An unexpected exception occurred in remote runner dummy_model: [500] Internal Server Error
2022-12-06T213320+0000 [INFO] [api_server:2] 172.17.0.1:58400 (scheme=http,method=POST,path=/dum_prediction,type=text/plain,length=2) (status=500,type=application/json,length=2) 78.677ms (trace=45f0871eb43b33f58cc42bba8e228540,span=8c2fd2b1351ad1ec,sampled=0)
2022-12-06T213503+0000 [ERROR] [runnerdummy model8] Exception in ASGI application
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py”, line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 78, in call
return await self.app(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/message_logger.py”, line 86, in call
raise exc from None
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/message_logger.py”, line 82, in call
await self.app(scope, inner_receive, inner_send)
File “/usr/local/lib/python3.9/site-packages/starlette/applications.py”, line 125, in call
await self.middleware_stack(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 184, in call
raise exc
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 162, in call
await self.app(scope, receive, _send)
File “/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/asgi/__init__.py”, line 487, in call
await self.app(scope, otel_receive, otel_send)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http/instruments.py”, line 293, in call
await self.app(scope, receive, wrapped_send)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http/access.py”, line 126, in call
await self.app(scope, receive, wrapped_send)
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 79, in call
raise exc
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 68, in call
await self.app(scope, receive, sender)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 706, in call
await route.handle(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 276, in handle
await self.app(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 66, in app
response = await func(request)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/runner_app.py”, line 270, in _request_handler
payload = await infer(params)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/marshal/dispatcher.py”, line 166, in _func
raise r
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/marshal/dispatcher.py”, line 397, in outbound_call
outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/runner_app.py”, line 255, in infer_batch
payloads = AutoContainer.batch_to_payloads(
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 475, in batch_to_payloads
return container_cls.batch_to_payloads(batch, indices, batch_dim)
File “/usr/local/lib/python3.9/site-packages/simple_di/__init__.py”, line 139, in _
return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 341, in batch_to_payloads
batches = cls.batch_to_batches(batch, indices, batch_dim)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 314, in batch_to_batches
return [batch[indices[i] : indices[i + 1]] for i in range(len(indices) - 1)]
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 314, in <listcomp>
return [batch[indices[i] : indices[i + 1]] for i in range(len(indices) - 1)]
TypeError: ‘int’ object is not subscriptable
Liwen Huang
12/06/2022, 9:41 PMis it because
Copy code
@dummy_service.api(input=Text(), output=JSON())
def dum_prediction(input_text: str) -> dict[int]:
return dummy_runner.run(input_text)s
sauyon
12/06/2022, 9:41 PMAh, it looks like you've enabled batching for your runnable and that doesn't work with your input type.
sauyon
12/06/2022, 9:42 PMWhat's the type of
dummy_runnerl
Liwen Huang
12/06/2022, 9:42 PMdummy runner is a dummy function that just return something
s
sauyon
12/06/2022, 9:42 PMHave you set
batchable=Truel
Liwen Huang
12/06/2022, 9:42 PM Copy code
import bentoml
def dumdum_model(input_text) -> int:
return 888888
# `save_model` saves a given python object or function
if __name__ == "__main__":
saved_model = bentoml.picklable_model.save_model(
'dummy_model',
dumdum_model,
signatures={"__call__": {"batchable": True}}
)
print(f"Model saved: {saved_model}") #dummy_model:zr64nfdvtoikrlg6Liwen Huang
12/06/2022, 9:43 PMno, I do not know how to do that
s
sauyon
12/06/2022, 9:43 PMRight, you've set
"batchable": TrueFalsel
Liwen Huang
12/06/2022, 9:43 PM Copy code
from <http://bentoml.io|bentoml.io> import JSON
from <http://bentoml.io|bentoml.io> import Text
BENTO_MODEL_TAG = "dummy_model:zr64nfdvtoikrlg6"
dummy_runner = bentoml.picklable_model.get(BENTO_MODEL_TAG).to_runner()
dummy_service = bentoml.Service("dummy_service", runners=[dummy_runner])
@dummy_service.api(input=Text(), output=JSON())
def dum_prediction(input_text: str) -> dict[int]:
return dummy_runner.run(input_text)s
sauyon
12/06/2022, 9:44 PMWe should probably do some sort of validation on the type annotations if they exist in order to catch situations like this.
l
Liwen Huang
12/06/2022, 9:44 PMbut the weirdest thing is before the containerize step, I am able to serve it as it is
s
sauyon
12/06/2022, 9:44 PMInside of your model training code, you've set signatures to be
{"__call__": {"batchable": True}}{"__call__": {"batchable": False}}sauyon
12/06/2022, 9:44 PMIt will work in development mode because we don't enable batching in development mode.
l
Liwen Huang
12/06/2022, 9:45 PMok
Liwen Huang
12/06/2022, 9:45 PMso change this and rebuild
s
sauyon
12/06/2022, 9:45 PMYou'll have to retrain the model too, in this case.
l
Liwen Huang
12/06/2022, 9:45 PMyep, gotcha
Liwen Huang
12/06/2022, 9:50 PMok I got the same error
Liwen Huang
12/06/2022, 9:50 PMTraceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/uvicorn/protocols/http/h11_impl.py”, line 407, in run_asgi
result = await app( # type: ignore[func-returns-value]
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/proxy_headers.py”, line 78, in call
return await self.app(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/message_logger.py”, line 86, in call
raise exc from None
File “/usr/local/lib/python3.9/site-packages/uvicorn/middleware/message_logger.py”, line 82, in call
await self.app(scope, inner_receive, inner_send)
File “/usr/local/lib/python3.9/site-packages/starlette/applications.py”, line 125, in call
await self.middleware_stack(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 184, in call
raise exc
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/errors.py”, line 162, in call
await self.app(scope, receive, _send)
File “/usr/local/lib/python3.9/site-packages/opentelemetry/instrumentation/asgi/__init__.py”, line 487, in call
await self.app(scope, otel_receive, otel_send)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http/instruments.py”, line 293, in call
await self.app(scope, receive, wrapped_send)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http/access.py”, line 126, in call
await self.app(scope, receive, wrapped_send)
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 79, in call
raise exc
File “/usr/local/lib/python3.9/site-packages/starlette/middleware/exceptions.py”, line 68, in call
await self.app(scope, receive, sender)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 706, in call
await route.handle(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 276, in handle
await self.app(scope, receive, send)
File “/usr/local/lib/python3.9/site-packages/starlette/routing.py”, line 66, in app
response = await func(request)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/runner_app.py”, line 270, in _request_handler
payload = await infer(params)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/marshal/dispatcher.py”, line 166, in _func
raise r
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/marshal/dispatcher.py”, line 397, in outbound_call
outputs = await self.callback(tuple(d for _, d, _ in inputs_info))
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/runner_app.py”, line 255, in infer_batch
payloads = AutoContainer.batch_to_payloads(
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 475, in batch_to_payloads
return container_cls.batch_to_payloads(batch, indices, batch_dim)
File “/usr/local/lib/python3.9/site-packages/simple_di/__init__.py”, line 139, in _
return func(*_inject_args(bind.args), **_inject_kwargs(bind.kwargs))
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 341, in batch_to_payloads
batches = cls.batch_to_batches(batch, indices, batch_dim)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 314, in batch_to_batches
return [batch[indices[i] : indices[i + 1]] for i in range(len(indices) - 1)]
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/container.py”, line 314, in <listcomp>
return [batch[indices[i] : indices[i + 1]] for i in range(len(indices) - 1)]
TypeError: ‘int’ object is not subscriptable
2022-12-06T214841+0000 [ERROR] [api_server:6] Exception on /dum_prediction [POST] (trace=66d1239a90d0b26266799dea1f3518a2,span=a791f440630b5a4a,sampled=0)
Traceback (most recent call last):
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/server/http_app.py”, line 336, in api_func
output = await run_in_threadpool(api.func, input_data)
File “/usr/local/lib/python3.9/site-packages/starlette/concurrency.py”, line 41, in run_in_threadpool
return await anyio.to_thread.run_sync(func, *args)
File “/usr/local/lib/python3.9/site-packages/anyio/to_thread.py”, line 31, in run_sync
return await get_asynclib().run_sync_in_worker_thread(
File “/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py”, line 937, in run_sync_in_worker_thread
return await future
File “/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py”, line 867, in run
result = context.run(func, *args)
File “/home/bentoml/bento/src/service.py”, line 13, in dum_prediction
return dummy_runner.run(input_text)
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py”, line 46, in run
return self.runner._runner_handle.run_method( # type: ignore
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner_handle/remote.py”, line 246, in run_method
anyio.from_thread.run(
File “/usr/local/lib/python3.9/site-packages/anyio/from_thread.py”, line 49, in run
return asynclib.run_async_from_thread(func, *args)
File “/usr/local/lib/python3.9/site-packages/anyio/_backends/_asyncio.py”, line 970, in run_async_from_thread
return f.result()
File “/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 446, in result
return self.__get_result()
File “/usr/local/lib/python3.9/concurrent/futures/_base.py”, line 391, in __get_result
raise self._exception
File “/usr/local/lib/python3.9/site-packages/bentoml/_internal/runner/runner_handle/remote.py”, line 201, in async_run_method
raise RemoteException(
bentoml.exceptions.RemoteException: An unexpected exception occurred in remote runner dummy_model: [500] Internal Server Error
2022-12-06T214841+0000 [INFO] [api_server:6] 172.17.0.1:58416 (scheme=http,method=POST,path=/dum_prediction,type=text/plain,length=4) (status=500,type=application/json,length=2) 75.113ms (trace=66d1239a90d0b26266799dea1f3518a2,span=a791f440630b5a4a,sampled=0)
s
sauyon
12/06/2022, 9:50 PMHm, that's strange. Did you update the model tag in your bento?
l
Liwen Huang
12/06/2022, 9:51 PMoh ahahaha, I forgot
Liwen Huang
12/06/2022, 9:53 PMwe can not use “lastest” in the model tag?
s
sauyon
12/06/2022, 9:54 PMI'm not quite sure, but I think if you don't include a version in the tag it gets the latest one by default. We don't recommend that for production, though, for obvious reasons.
l
Liwen Huang
12/06/2022, 9:55 PMyeah
22 Views