Hi all, I'm debugging some performance issues we've been seeing since upgrading to bentoml 1.3 from 1.1 I'm hoping to understand better the expected behavior of the

to_async

method of bentoml services. I ran a couple load tests locally with the following code:

import bentoml
import time
import asyncio
import random


@bentoml.service(workers=4)
class LongRunningService:
    @bentoml.api
    async def long_running_task_async(self, n: int):
        await asyncio.sleep(n)
        return random.random()

    @bentoml.api
    def long_running_task(self, n: int):
        time.sleep(n)
        return random.random()


@bentoml.service(
    workers=1
)
class svc:
    long_running_service = bentoml.depends(LongRunningService)

    @bentoml.api
    async def predict(self, inputs: str) -> str:
        # x = await self.long_running_service.to_async.long_running_task(n=3)
        x = await self.long_running_service.long_running_task_async(n=3)
        return f"{x}"

When I use the

x = await self.long_running_service.long_running_task_async(n=3)

function, all requests are returned within 3 seconds because long_running_task_async utilizes

asyncio.sleep(n)

which is non blocking. Conversely, when I use the

x = await self.<http://long_running_service.to|long_running_service.to>_async.long_running_task(n=3)

, requests begin stacking up in the internal queue and the response time gradually grows. Is this expected behavior? Based on the docs, I would've assumed that

to_async

would've turned the function into non-blocking code like the

long_running_task_async()

function .

it allows one to calls a dependent service's sync method as an async function so that it does not block the main service, which is

svc

in your case. However in the

LongRunningService

,

time.sleep

is still a blocking code, so as requests stacks up,

LongRunningService

eventually gets blocked because within the service it is still a synchronous function. CC : @Frost Ming for clarification

It is turned to async at the client side, but the server still blocks at

time.sleep()

due to the default thread capacity=1

Thank you for your response. That makes sense. We are in the process of updating our services from bentoml 1.1 to 1.3. So far we have seen significantly increased CPU usage in the updated version for our non-gpu models so we are trying to determine what is going on. In 1.1, we utilized a main server (equivalent of svc above) and a runner with our model in it (equivalent of LongRunningService above). In 1.3, we have two services representing the main entry point and a service containing our model. We've tried increasing throughput by increasing the number of workers on the model service but that actually ended up reducing our performance and caused errors. Also, our model services have batchable=False in both 1.1 and 1.3 Do you have any general guidelines for optimizing performance/throughput/latency on cpu-only models?