[2] the default runner workers is set to num of cpu cores on instance. If you have 4 cpu cores there will be 4 photocopies of runner, each taking GPU. You can limit the config file parameters. https://docs.bentoml.org/en/latest/guides/configuration.html essentially changing

bentoml_configuration.yaml

version: 1
api_server:
  workers: 1

will ensure that single runner is being executed. But the internal request scheduling is performed better if multiple runners are being executed. Also have a look at 📜 https://docs.bentoml.org/en/latest/guides/scheduling.html

[7] if you are asking from json input perspective (keeping some default values in json)

from <http://bentoml.io|bentoml.io> import Image, JSON

class JsonArgs(BaseModel):

prompt: str

height: t.Optional[int] = 1024

width: t.Optional[int] = 1024

class Config:

extra = "allow"

sample = JsonArgs(prompt="custom text message")

then for service endpoint:

@svc.api(input=JSON.from_sample(sample), output=Image())