Hello! While testing a simple API, I noticed regular freezes lasting a few seconds. When I enable adaptive batching, these freezes occasionally occur around the updates of the dynamic batching optimizer parameters. In some of my tests, the API remains frozen for up to 10% of the total time, which is significant. Are there any known issues that could cause an API to freeze like this?

these freezes occasionally occur around the updates of the dynamic batching optimizer parameters

How did you observe that?

Hello, i did a few test in order to reproduce this issue. It ends up not being related to adaptive batching, this looks like a limitation of bentoml. Here is a small example to reproduce this issue. Initially, this example handles the input query with no issues, then the latency start to increase until it reaches multiple seconds. I only tried this script for a few minutes, i don't know if the latency ends up being stable after a long period of time.

import bentoml

@bentoml.service(
    traffic={"timeout": 10} # "concurrency": 1 gives similar results
)
class DummyAPI:

    @bentoml.api
    def dummy(self, text: str) -> str:
        return text

Using artillery, i measure that this small example cannot handle more than 100 requests per second (the latency eventually increases) while a low rate of 20 requests per second works just fine. Do you have an idea of the maximum throughput that bentoml can handle ? Maybe i need extra arguments in order to increase it ?

cc @Frost Ming could you help verify this?

Tested with locust and the RPS keeps at 600+ without obvious drop

@Lounès Lounis could you share a bit more about your setup?