This message was deleted.

Is the model served on GPU or CPU?

GPU

Could you use htop to inspect if the runner's CPU usage is exhausted? You can start the bentoml production server, then run htop, then press F4 and input

bentoml_cli.worker.runner

. This will isolate the runner process

As I mentioned, I have more than enough free resources 🙂

Ye I see, I'm asking because in some situation even though the GPU utilization is low, the runner has performance bottleneck at CPU part. But I can see that it's not the cause for your case

Do you have any suggestions on how to debug it?

have you tried increasing max latency?

So max latency of 1 second or 1000 ms is not good enough. I can increase the max latency but also increase the workers amount to make sure I am using up my resources efficiently?

These fails most likely happens at the beginning of the service. Because we use CORK algorithm (a variation of Nagle's algorithm[0]) to determine when to release a batch. We need to analyze historical data to optimizing the parameters of this algorithm in realtime, while at the beginning of serving, we don't have enough historical data, so some failures will happen. Could you try to do 6k requests and see if the failure rate drops to around 3%? [0]: https://en.wikipedia.org/wiki/Nagle%27s_algorithm

@larme (shenyang) Okay, I sent 11K requests first as a warm up. Max latency is defined as 10,000. I sent a total of 6,482 requests at a pace of 120 requests per second. I had 2,597 failed requests (40.06% fail rate) • When I sent 3K requests at a pace of 40 requests per second I had 0 failed requests. • When I sent 4,818 requests at a pace of 60 requests per second I had 460 failed requests (~9.55% fail rate)

what's the result of doing a warm up then send 6k requests at 40 rps? If the failure rate drop then that means at 40 rps the service is stable after some warmup. Then we can do test at 80 rps, 120 rps etc.

@larme (shenyang) Edited my message

ok I think when you do test at 120 rps, could you monitor the runner cpu usage again? Sorry I just want to make sure the bottleneck is not runner's cpu usage

Sure it will take 2 minutes

and what's the GPU utilization?

~65%

I'm not entirely sure but one possibility is that when runner is doing CPU works, the GPU is waiting for data input, so nearly half of the GPU resource is not utilized.

And when I increase the max latency I just have lots of requests that stay pending

yes, if the runner is overloaded, then pending requests will accumulate

@larme (shenyang) Really appreciate the help! Generally speaking it means that I can't fully utilize the batch functionality because the runner gets overloaded quite easily

It's something we will investigate. We already have some experimental patches to improve the performance and they will be merged in following releases

The bottom line is that HTTP 429 shouldn’t result in runner handle TypeError. HTTP 429 should be handled by the framework. We will try to reproduce internally.

What can I do in the meantime? I want to have the bento v1 used in production

I think the easiest thing to in the meantime is to use single-return value API functions; this seems like it's a bug in the multi-return code.

@sauyon Won't it be not using the batching?

You can continue to return a list, just not a tuple of return values.

How would that work? How can I prevent it from returning a tuple of return values?

What runner are you using right now?

Pytorch based

Is it a custom runner or the default BentoML one?

default

Hm, ok, let me see if I can write something up that should hopefully help in the short term; that might take me a little bit, though. Do your models normally return tuples?

My models return a list

Oh, I think that's another bug, then---we shouldn't be using the multiple return path at all :(

ye only tuple should trigger the multiple output path

This is the return value

def pytorch_predict(self, d):
    return [x.pandas().xyxy for x in self(d).tolist()]

Hm, well, I'm now very stumped. Would you be able to provide the result of

type(<your runner>.<method>.run(...))

?

@sauyon Of course I can. It will take a moment

Yeah, that definitely seems to be the case...

Because at the end of the day, I get the 429 while I am far from using up all my resources or the batch size limit I have defined on the runner

Let me just find which version we added multiple output support, maybe your code will work before we added that.

Would love to 🙂 Thanks

Ok, can you try 1.0.5 and see if that behaves any better?

Sure one sec

Nope, you should just be able to run it with 1.0.5.

I get the following exception when trying to load my application now: ValueError: expected API function to have arguments (files_stream, form_stream, [context]), got (files_stream, form_stream, ctx)

Ugh, that was a bug with multipart descriptors. Are you able to use a BentoML built from a commit?

Well I'd rather keep the version 1.0.6 that I have been using and I don't mind editing some of the code to make stuff work better 🙂

from bentoml._internal.runner.strategy import Strategy
from bentoml._internal.runner.strategy import DefaultStrategy
import math


class NxStrategy(Strategy):
    n = 2

    @classmethod
    def get_worker_count(cls, runnable_class, resource_request):
        return cls.n * DefaultStrategy.get_worker_count(
            runnable_class, resource_request
        )

    @classmethod
    def get_worker_env(
        cls,
        runnable_class,
        resource_request,
        worker_index,
    ):
        pseudo_worker_index = math.floor(worker_index / cls.n)
        return DefaultStrategy.get_worker_env(
            runnable_class, resource_request, pseudo_worker_index
        )


object.__setattr__(<your runner>, "scheduling_strategy", NxStrategy)

This is the code you can put into your service; it's a bit hacky because we don't really support custom strategies at the moment, at least for default runners, but it should hopefully work.

And this might solve my 429 issue?

That should spawn

n

times the number of runner processes that would normally be spawned by our default strategy; basically higher

n

should hopefully speed things up, as long as you have enough memory. The 429 error is something that happens when our runner can't keep up, for one reason or another, so spawning more runner processes should help with saturating your resources and avoiding that bottleneck.

Because it will override the scheduling strategy that is already defined in the runner?

In theory yes, but at the moment it appears that some pre- or post-processing code in your runner process is preventing it from reaching full utilization; my guess is there's some relatively heavy python which is slowing down the runner and preventing it from utilizing all resources available.

When you say heavy pre- or post-processing code in my runner process that prevents reaching full utilization, you mean in my model's inference or my api endpoint?

Inside the model's inference; the API server should be spawning enough processes for that not to be a problem