BentoML

In `vLLM`, you can shard a model across multiple GPUs using engine arguments `-pp` or `-tp`.  Does `openllm` support those `vLLM` command line options?

Yes. But at present you need to follow the <https://github.com/bentoml/OpenLLM/blob/main/DEVELOPMENT.md|developer guide> to add custom recipe to your own repo. This is a little bit complex but you can take the full control and share your models(bentos) to others.
You can control the pp and tp in `engine_config:`

We plan to make it easier in the next version.

Thank you <@UQF2L5GQ0>, but I don't quite understand your response.  Our product deploys models that use BentoML in Kubernetes, so we don't have control over how the user created the Bento or the OpenLLM model.  Basically, if a user wants to deploy an OpenLLM model from HuggingFace, and the model is too large to fit into a single GPU, how do we tell OpenLLM to shard a model across multiple GPUs (split a model into parts and distribute them across several GPUs)?

<@UQF2L5GQ0> is what I'm asking for not possible today?  If it is not, may I enter a ticket to have this functionality added in a future release?

<@U067AJCFEQH>
Sorry for the late reply.
> we don't have control over how the user created the Bento or the OpenLLM model
Got you point. If we don't have control over the bento or model itself, it is impossible to override the model's behavior from outside with the current openllm.

openllm 0.6 is just designed as a launcher of bentos. Developers can define working configs in the bentos.

For example, you can maintain your own openllm model repo with:
```llama3:70b-4bit-a100
service_config:
   gpu: 2
   gpu_type: nvidia-tesla-a100
engine_config:
   tensor_parallel_size: 2```

```llama3:70b-4bit-a10g
service_config:
   gpu: 8
   gpu_type: nvidia-tesla-a10g
engine_config:
   tensor_parallel_size: 8```
With `openllm repo add <your_repo>` , openllm can suggest correct model according to the resources.