BentoML

Hi @everyone

I am *new to BentoML*, I was going through OpenLLM for LLM Deployment on our Kubernetes. Now we have *trained Mistral LLM on custom data* case-specific training.

I see to start openllm for mistral, I just need to execute openllm serve mistral:7b but this serves standard Mistral from Huggingface.

Now my question is how *I can serve my trained LLM on OpenLLM*. I tried to search docs but didn't get a clear answer so asking here. Sorry if I am asking a very basic question

Regards
Nabarun Barua

Hi <@U07BCBAMDPF> - here’s the instruction for building a custom repository in OpenLLM, which allow users to bring in custom models

<https://github.com/bentoml/OpenLLM?tab=readme-ov-file#set-up-a-custom-repository>

we are working on a new feature that would make this process a lot easier

Thanks, <@UKB4CLKP1>. I'll be waiting for the new feature. In the meantime, I'll try the approach shared.

Hi <@UKB4CLKP1>, I forgot to ask you if you have any tentative plans for when your team will release this new feature.

Thanks in Advance :slightly_smiling_face:

<@U01BMCJ29LN> <@UQF2L5GQ0> any estimates when this will be released?

This is already possible. U can just set up a custom repo and then use openllm repo asd 

<@U01BMCJ29LN> - this is the new CLI for serving fine tuned model

Hi, <@U07BCBAMDPF> sorry I'm just back from vacation. The new feature should arrive in two weeks, sooner or later. We know it is already possible, but we are trying to make it as easy of use as possible.

Thanks <@UQF2L5GQ0> &amp; <@UKB4CLKP1>. Looking forward for to it.