BentoML

Yes, they do, at least in production mode.

```import bentoml
from <http://bentoml.io|bentoml.io> import NumpyNdarray

model = bentoml.pytorch.get(
    'model:latest').to_runner()
svc = bentoml.Service(
    'shrimp-predictor', runners=[model])

@svc.api(input=NumpyNdarray(), output=NumpyNdarray())
async def func(data):
    for _ in range(100):
        await model.async_run(data)
    return data```


```from torch import nn

class Net(nn.Module):
    def forward(self, x):
        print(f'Model input {x.shape}')
        return x```

```import bentoml

bentoml.pytorch.save_model('model', Net(), signatures={'__call__': {'batchable': True, 'batch_dim': 0}})```

```from bentoml.client import Client
import numpy as np

bento_client = Client.from_url('<http://localhost:7777>')
data = np.array([[1]])
bento_client.func(data)```

```pyenv exec python -m bentoml serve --production draft:svc -p 7777```


It seems like you're trying to run await in a for loop; that will just run each request in series.

I think in python to run those tasks, you should use `create_task` instead, and then await those tasks.