This message was deleted.
# ask-for-help
s
This message was deleted.
j
BentoML is flexible enough for you do things like this! I would say you are free to do whatever you need
c
Nice! However, looping over a function does not work in python as you can not define the same function multiple times. So how does Bento solve this?
j
right, what are your expected behaviour from the user POV? would you expect to have 80 endpoints with different routes? does the naming of the routes matter
c
Exactly. So i want the same function basically, but with 80 different route names and with 80 different model runners.
j
right, this is probably not idiomatic BentoML code, but you can try by using a closure and editting the
__name__
and
__qualname__
Copy code
def base_function():
    print("Hello from the base function")

def create_function_with_closure(new_name):
    def closure():
        return base_function()

    closure.__name__ = new_name
    closure.__qualname__ = new_name
    return closure

# Create a new function with a dynamic name
dynamic_func = create_function_with_closure("dynamic_func")
print(dynamic_func.__name__)  # Outputs: dynamic_func
dynamic_func()  # Calls the base_function
c
I will try this!
j
let us know if that works! this is a good example for advanced use cases that we can use as example!
c
Great. So the api annotation goes with the base function? Only thing i do not know if it will work os selecting the correct runner in the base function, i somehow ould need to pass an index or something lile that from the list of runners
As a feature in general it would be nice to have something like this in bento: svc.add_api(input=Image(), output=JSON(), route=”route1“, base_func=base_func, args_func=None, async=True)
t
Hi Chris, we were facing similar situation and we used Jinja template to generate the service.py file at run time. Jinja allows looping / if else / condition checking functionality. So you can use a skeleton code as template and create a jinja block. based on the user selected model you can render the endpoint.
Apart from that please have a look at examples/custom_runner --> yolov5 example since it shows how you can load the model inside the init which also allows to decrease the bento created docker image size.
c
thanks for the hint. i did it in a different way, and the bento build with the CLI works:
Copy code
def setMethod(anomaly_runner: Runner, preprocess: Preprocess):
    async def predict_image(f: Image) -> list[InferenceDefect]:
        # do stuff and return
        return result
    return predict_image

for idx, (runner, preprocess, fullname) in enumerate(zip(runner_list, preprocess_list, model_names)):
    path = f"predict_image/{'/'.join(fullname.split('_')[:4])}"
    fn = setMethod(anomaly_runner=runner, preprocess=preprocess, index=idx)
    _api = InferenceAPI[IOType](  #  Reuse code from bentoml Decorator
        name=first_not_none(f"predict_image_{fullname}"),
        user_defined_callback=fn,
        input_descriptor=Image(),
        output_descriptor=JSON(),
        doc=None,
        route=path,
    )
    if _api.name in svc.apis:
        raise BentoMLException(
            f"API {_api.name} is already defined in Service {svc.name}"
        )
    svc.apis[_api.name] = _api
However. I am now trying to do the build not with the CLI. And i thought i could simply to something like
Copy code
bentoml.bentos.build(svc, models=model_names) # svc is the bentoml.Service object
However, it seems the build needs a .py file with all the code and just passing the Service object does not work? Is this correct? I do not know if generating python code to build a service is really a nice solution for such a automatized process... @Talha Yousuf is that the reason why you used Jinja?
oh okey as we need to store the files in bento i think this makes sense. i think the missing configurations i need, i can also provide with hydra and my code solution from above
j
ah yes, in our other projects OpenLLM and OpenDiffusion, we do templating for the service files as well. would you guys like to share you complete example in #i-made-this ? this could be an excellent advanced use case for the other uses in the community
👍 1
c
@Jian Shen Yap I am working on a small change in the service class of bentoml and also providing a new example. i will post the pull request soon, so you can take a look, how i solved this issue 🙂
@Talha Yousuf maybe this is also interesting for you - i did not want to use jinja :D
j
This is awesome! i think it may be difficult for us to merge this into the main codebase for the moment because we just released 1.2 bentoml and that has huge changes on how we look at services. nonetheless, i'll let @Frost Ming to have a look soon
👍 1