https://bentoml.com logo
Join Slack
Powered by
Hello all, I have been using bentoml 1.1.11 and j...
# ask-for-help
d
Hello all, I have been using bentoml 1.1.11 and just updated to bentoml 1.2.9, and I realized that now it's a completely different service. The way you define a service, runnable, and the api inputs/outputs, so many things are changed, and they are not backward compatible. Now I'm getting errors from pydantic, and for some reason bentoml build command tries to inject the new version to my requirements.txt. Is there a migration document that I can get some help?
❤️‍🩹 2
🙏 2
e
As a customer, I can say there's good and bad here (and I really like BentoML!!!) 1. The bad: Yeah, we felt blindsided by the switch from 1.1 to 1.2 -- genuinely the framework feels more confusing. a. I was okay having services and runners be separate and forcing everything to be in a class feels like a step backward. My opinion: you deviate from FastAPI's API design at your peril--so you should have strong reasons for doing so. I'm sure they had reasons, e.g. attempting to simplify, but the result is an API I don't love. b. It also introduced many bugs as you pointed out, c. and (for the second time in BentoML's history) rendered the existing world of blogs, tutorials, trained LLMs, etc. close to useless. I miss 1.1 and don't yet see a way 1.2 is better. d. As a new customer, I think there's a strong argument that the change from 1.1 to 1.2 set us back at least a month on seeing ROI from the tool. (BentoML team has had increased workload responding to 1.2 bugs found by their users, we had to ask them 3x the support questions for code examples since docs weren't out, etc.) 2. The good: At the same time: BentoML is working hard to take away some of these rough edges. a. I've paid close attention to the docs and have seen new guides appear every day. b. They've genuinely listened to our complaints. For example, they are scoping out what it would take to bring back 
runner.async_run
and 
runner.run
. TBD on 
bentoml.<framework>.save_model(...)
. c. When provided with specific code examples of errors or general issues with the API design, they've pushed fixes fairly quickly. E.g. there were more issues with Pydantic at the beginning, at least ours were solved quickly. d. Yatai 2.0 will be under the Apache 2.0 license 🎉. Really generous of them. I still believe in the tool and the team behind it--enough that me and some folks where I work are contributing this VS Code extension--and hopefully Yatai 2.0--for BentoML in our personal time. I think 1.2 is less ergonomic and was harmful to the community, but I still don't see any other tool that comes close to BentoML's features and workflow. I'm having to give this exact pitch to our CTO today to explain some of our delays, hence taking the level of detail 😆
➕ 4
🙏 2
a
Thanks for the detailed write up, Eric. I’m considering adopting BentoML. Had been working on this with the 
1.1.x
series, and definitely agree that 
1.2.x
seems like a bit of a step backwards, particularly with respect to the breaking changes and documentation/tutorials being outdated. Also agree that I don’t love the class-based approach in the 
1.2.x
series. Not sure what the design considerations are there, though.
🙏 2
💡 1
m
@Eric Riddoch I agree with your take, both good and bad. I'm also excited by the possibility of a Yatai 2.0. Hope to contribute to it in my personal time, but tbh am finding it difficult right now to make that time. Fwiw, @Chaoyu has assured me (in a DM) that 
1.1.x
will continue to receive critical vulnerability & bug fixes. My plan is to stick with 
1.1
as long as it's feasible. I'm sad to miss out on some of the new functionality offered in 
1.2
, but it's just too disruptive for me to make that migration at this time. Love BentoML in general, don't love the too frequent backwards incompatible changes, and especially the lack of transparent communication about what's coming.
💡 1
➕ 1
🙏 2
v
Very interesting discussion as my team hasn't yet made the switch. I feel less worried about switching to classes but I would agree that the release felt undercooked (and that's why we haven't switched to 1.2). May I ask where you found the info about yatai 2.0? Last I heard, yatai was "done" and in maintenance mode.
🙏 1
m
Here is a link to the slack announcement about yatai 2.0: https://bentoml.slack.com/archives/CK8PQU2JY/p1709063701552909
s
Thank you all for the valuable feedback and engaging discussion. We’ve gained a lot of insights and have taken time to reflect. We must acknowledge that the planning and release of BentoML 1.2 were not as transparent as they should have been, and we failed to provide the community with an opportunity to offer feedback on the design. This oversight is on us, and we recognize that it’s not the way to run a large open-source community. We’ve taken your feedback to heart and are committed to improving our processes. More in threads.
❤️ 3
gratitude thank you 2
To address the concerns raised, we would like to share the rationale behind the design of BentoML 1.2. During the design process, we debated between class-based and function-based APIs. Given that BentoML is primarily focused on serving models, it is crucial to provide the ability to manage service lifecycles and states. This includes service initialization, where models are loaded, and state management, which allows APIs to access initialized models in an intuitive and developer-friendly manner. We considered service initialization in the global scope, similar to FastAPI. However, we concluded that the global space was not suitable for initialization for the following reasons: 1. Initialization logic for model loading is typically resource-intensive, so we want to avoid invoking such logic when the module is imported, for example, by 
bentoml build
. 2. With multiple services, each service requires individual initialization. A shared global scope would not suffice. We also explored the possibility of supporting service initialization in a functional manner, such as decorating a function with 
@svc.on_startup()
. The challenge arose in managing states. Since models are common states shared across different APIs, they need to be accessed in an intuitive and developer-friendly way. With a functional approach, states would have to be managed similarly to FastAPI, with fields set in 
app.state
. However, managing models with 
app.state
is not explicit, as the syntax needs to clearly indicate to users that models will be treated specially during building and serving. Additionally, this approach does not support type hints in IDE auto-completion. As a result, we opted for a class-based API. We believe that the 
__init__
constructor in classes most intuitively represents service initialization to Python developers. Moreover, classes most meaningfully represent services, so when we have multiple services, we simply define multiple classes. Interactions between different services are then modeled as function calls across different classes.
👍 2
The decision to remove the framework runners was driven by the observation that there are increasingly more powerful libraries that accelerate model inference on top of the base ML framework, like vllm and trt-llm for running large language models. Choosing the appropriate library is highly dependent on the specific use case. As a generic framework, providing a one-size-fits-all solution was becoming increasingly challenging. We believed that the best approach was to introduce the framework in a flexible and non-intrusive way, allowing users to integrate with any framework and library of their choice. However, we realize that we may have swung from one extreme to the other. Thanks to your feedback, we recognized the importance of continuing to support framework runners. Framework runners were one of the main reasons our users chose BentoML, as they provided a convenient way to run classical ML models like those in XGBoost and Scikit-Learn. In response to this feedback, we will reintroduce framework runners in BentoML 1.2 as framework services. This will provide an easy way to turn models into a service without having to explicitly define the service class.
🎉 1
The BentoML 1.2 blog post will also address some of the design considerations not mentioned here. Overall, thank you for your feedback. We will update our progress on the action mentioned above.
a
Hi Sean, appreciate the detailed response and addressing some of the feedback. One question I have is what is the impact of the class-based approach on on runners vs. services/bentos. This also seems related to the decision to remove the framework-specific components of BentoML, so maybe that’s why it wasn’t clear to me what the impact was. My understanding in 
1.0.x
and 
1.1.x
was that a given model service could have one or more runners, one for each model that made up that service (or bento). From what I understand, the each runner ran in its own Python process (and all of these were independent of the Python process handling the web requests), or, if you were using the Yatai CRDs, the runners were actually running in separate containers (and could scale independently). How does that all work now in 
1.2.x
, and will the reintroduction of framework runners impact this at all?
s
Will the old documentation be updated to support these changes?? like examples? Never mind i just got it
a
Two followup questions to what I had above, as well: how does this interact with serving over gRPC and also how does this interact with other core BentoML features such as adaptive batching?
2 Views
Open in Slack
PreviousNext
Powered by