Hi, guys, Do you have any idea why api_serve is quite slow compared to the runner? api_server : ~1400ms runner: ~150ms this cause overall improvement in service's RPS

I have the same problem. I have a quite fast model, but somehow serving local with the bento serve command adds 3sec overhead (compared to 0,3sec inference with torch)

Do you have any alternative ways ? I'm thinking, I will serve my model with FastAPI.

Unfortunately not… i also create a fastapi server. What gave a little bit of a boost was reducing the number of workers on windows

Thank you for sharing I will test with my service and share a result to you.