Hi everyone, I have a question regarding service optimization. Are there any general guidelines for configuring the number of workers for api servers vs model runners? We have a simple service with a DistilBert model and we recently added a GPU which significantly increased performance. However, changing the parameters for number of workers on both the model runner and api server seemed to make very little difference at all. The only thing that seems to be increasing throughput at this time is adding more cpu cores

@Thomas Busath I'm not aware of any general guidelines, but FWIW we've seen some similar things. Increasing the number of API workers didn't initially increase throughput of requests to our server at all. After digging a bit more, we discovered that we were GPU bound. So, throwing more requests at the model by increasing API workers didn't help, but improving the model speed (e.g. via quantization, higher quality GPU, etc.) did. And once we did that, the API workers then became the limiting factor and we were able to see gains in throughput as we scaled up the # of API workers.