This message was deleted.

Hi! This is a feature we're actually currently missing, and we need to implement. For now, the best you can do is to use a custom scheduler:

from bentoml._internal.runner.strategy import Strategy
from bentoml._internal.runner.strategy import DefaultStrategy
import math


class NxStrategy(Strategy):
    n = 2

    @classmethod
    def get_worker_count(cls, runnable_class, resource_request):
        return cls.n * DefaultStrategy.get_worker_count(
            runnable_class, resource_request
        )

    @classmethod
    def get_worker_env(
        cls,
        runnable_class,
        resource_request,
        worker_index,
    ):
        pseudo_worker_index = math.floor(worker_index / cls.n)
        return DefaultStrategy.get_worker_env(
            runnable_class, resource_request, pseudo_worker_index
        )


object.__setattr__(<your runner>, "scheduling_strategy", NxStrategy)

Thank you, i will try