This message was deleted.

@Daniel Buades Marcos thanks for reporting it. Can you give us a quick reproducible if you have the time? We will debug it more and see where the issues are. cc @sauyon @Sean

@Bo Sure, I’ll get back to you before EOD 👌

appreciate it

2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=480) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75032) 1903.326ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=64ced498b005c746,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1224) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75787) 1904.100ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=5087201a4a06e264,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=492) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75046) 1904.737ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=456dae7a71e4b5bf,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1532) (status=200,type=application/vnd.bentoml.DefaultContainer,length=76086) 1905.515ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=f8bfd34c6d1617e8,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=696) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75251) 1906.179ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=9ab8aa7462e23643,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1521) (status=200,type=application/vnd.bentoml.DefaultContainer,length=76076) 1906.743ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=0036dc8d40703220,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=862) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75417) 1907.341ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=6320c0fb895b7054,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=718) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75271) 1914.927ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=d6f9d71229c4bcb3,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1348) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75901) 1915.902ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=6a20fcbeddaa54f9,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1368) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75921) 1917.026ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=31fe63b333e0f0a2,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=473) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75029) 1917.900ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=26f17050c9a5022c,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1558) (status=200,type=application/vnd.bentoml.DefaultContainer,length=76111) 1919.082ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=bfae213aa39ccc5b,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=844) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75397) 1920.276ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=23208081c8a41744,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=585) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75138) 1921.195ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=d57f2535decbf1d1,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=716) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75269) 1922.615ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=5a974b28ad61d584,sampled=1)
2022-10-12T10:45:52-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1061) (status=200,type=application/vnd.bentoml.DefaultContainer,length=75614) 1923.695ms (trace=cbd20a373f43fcf901e74f8f739645c8,span=cfd0eb51a1f6af03,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=834) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 15.822ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=f288261c31128489,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1284) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 16.394ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=b8bb092b94b96681,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1153) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 16.489ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=2239dfe3b5fda7a8,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=727) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 15.766ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=58e14c6a000bc389,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=995) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.383ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=520fe42aa75c0d85,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=632) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.268ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=59193e18af74f4da,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=663) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.286ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=46b8e449b6c1b213,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=943) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.292ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=389ca68fce858fab,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=706) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.309ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=b8905a71e9f938b7,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1248) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.365ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=7377d5c820aafd2e,sampled=1)
2022-10-12T10:47:06-0400 [INFO] [runner:ner:1] _ (scheme=http,method=POST,path=/relations,type=application/octet-stream,length=1061) (status=200,type=application/vnd.bentoml.multiple_outputs,length=136) 13.003ms (trace=fd52be56785bd2ea2d8b1305bc35d226,span=987f5266f8342c24,sampled=1)
2022-10-12T10:47:06-0400 [ERROR] [api_server:ner-aggregator:6] Exception on /api/v1/relations [POST] (trace=fd52be56785bd2ea2d8b1305bc35d226,span=42740386bfc39f62,sampled=1)
Traceback (most recent call last):
  File "/Users/dbuades/Library/Caches/pypoetry/virtualenvs/search-model-server-3GLlreJ3-py3.9/lib/python3.9/site-packages/bentoml/_internal/server/http_app.py", line 311, in api_func
    output = await api.func(input_data)
  File "/Users/dbuades/repos/search-model-server/models/ner_aggregator/service.py", line 101, in relations_single
    return await handler_single(
  File "/Users/dbuades/repos/search-model-server/models/ner_aggregator/__init__.py", line 46, in handler_single
    ner_results, name_matcher_results = await asyncio.gather(ner_results, name_matcher_results)
  File "/Users/dbuades/repos/search-model-server/models/ner_aggregator/__init__.py", line 121, in _ner_wrapper_single
    ner_results = await asyncio.gather(*coroutines)
  File "/Users/dbuades/Library/Caches/pypoetry/virtualenvs/search-model-server-3GLlreJ3-py3.9/lib/python3.9/site-packages/bentoml/_internal/runner/runner.py", line 51, in async_run
    return await self.runner._runner_handle.async_run_method(  # type: ignore
  File "/Users/dbuades/Library/Caches/pypoetry/virtualenvs/search-model-server-3GLlreJ3-py3.9/lib/python3.9/site-packages/bentoml/_internal/runner/runner_handle/remote.py", line 188, in async_run_method
    return tuple(AutoContainer.from_payload(payload) for payload in payloads)
TypeError: 'Response' object is not iterable

@Bo here is an excerpt from the logs, you can see how at some point, it changes to

multiple_outputs

I think this is the same issue that @Jiang was resolving earlier; it's a bug we introduced while reworking some runner code where if the runner is overloaded, it will send back the wrong response. Because the client doesn't know how to handle it, that's the error that you see. For right now, all that that error means is that the runner server is overloaded.

Thanks for your input @sauyon ! It looks like that might be the case, see the attached screenshots from a locust test I just did: 6/10 requests fail at the beginning of the test but then when the test is done only 15/5590 have failed. For those that fail (at the beginning and in the middle of the test), I can see in the logs that is because of the

multiple_outputs

type. Is it expected that the runner is overloaded in the first queries? Seems like this part might be a different bug.

Ah, yeah, requests tend to fail at the beginning because the runner's dispatcher is trying to learn how long requests take; it's something that we almost certainly want to fix eventually, but for now the recommendation is to throw a few sample requests at the server to warm it up every time.

@sauyon @Bo Thanks, in that case that confirms that this is what we are seeing here! Do you have an example of a custom readiness probe? I’d like to add a few queries to the readiness probe so the runner is already warm up when we start sending queries to the service.

great. Good to hear!