BentoML

Is there a way to have a dictionary of models (we have one model per client id)? If so, is there a way to control micro batching to be done at client id level? It wouldn't make sense to microbatch together requests from different clients.

Assuming all those models fits in one node in one service, you can try dynamically add APIs for each model during initialization, that way each model will be batched separately 

You can use the model ID as part of the request path instead of in the request body

Yeah, I was imagining something like this. I'll try it out. Thanks for the help, <@UKB4CLKP1>

<@UKB4CLKP1>, is there an API do create multiple instances of a given service like that? I was thinking about having the service class with the client_id as input for the constructor and then initializing multiple instances of it. I guess I wouldn't use the decorator in this case, right?

you can use the “workers” parameter to set how many instances of a service class will be initialized at runtime, each instance will get assigned a different worker id 

Check out the examples here: <https://docs.bentoml.com/en/latest/guides/workers.html|https://docs.bentoml.com/en/latest/guides/workers.html>

They can’t take different parameters for the constructor, but you can perhaps use the worker id to do something similar 

Thanks for the help, <@UKB4CLKP1>. But I'm still not sure how to do this. I was imagining a way in which, at startup, I have a list of client_ids with their respective models. I'd like to create one service per model and then redirect the request accordingly. If I just create one service with multiple workers, I don't know how I could define which instance is responsible for each client id and how to invoke the correct worker according to the client id.

I can't find any example in the documentation on how to use path parameters either. For example: `/{client_id}/predict`. I'd like to have the client_id como through the path and the rest come from the body. Would I need to use BentoML together with FastAPI for that?