Does anybody know if it's possible to run different runners in different virtual envs? Such that we could have different versions of a requirement (e.g.

numpy

) for different runners.

no that is currently not possible. In that case you may want to build different bentos

@Jian Shen Yap gotcha, thanks! Any idea if that's on the roadmap? I'm not sure exactly what that setting up multiple bentos would look like for us.. can you perhaps help me think through this? Or check my understanding below here? We've got 8 models spread across 2 dependency steps. Models in the 1st dependency step can run in parallel and models in the 2nd dependency step can run in parallel, but models in the 2nd dependency step depend on results in the 1st. Currently we're running all models on their own runner in a single Bento. To do this with multiple bentos, we'd have to set up 8 bentos, one for each model. And then each bento would have an endpoint to hit that would expose the model, but we'd have to manage communication between all of those endpoints - is that right?

Hi @Sean Sall! I'm actually tracking this feature as it's asked for every now and then. Would you mind getting on a quick call with me to walk me through your use case?

Hi @Tim Liu! Definitely happy to get on a call. If it's worth anything, I do believe my colleague Sudeep Ghimire has maybe chatted with you about this before? But regardless, I'm happy to. I'm still curious at how to achieve this with multiple Bentos if y'all have any documentation on that front!

oh... ha, yes he's one of the people that I talked to about this and have his feedback recorded

Hehe, I thought he had mentioned it. Happy to still give my perspective on our individual company over a call if that's helpful. But also, FWIW my TLDR is that the setup at the company Sudeep and I are at is a similar setup / problem space to the 2 prior companies I've been at in terms of deploying ML (and all these companies were in regulated spaces). Each of the companies had multiple models to deploy, and the models that were developed first were developed with earlier versions of dependencies than the later models, resulting in conflicts if you have only a single environment. Forced updating of the dependencies for the earlier models just because we want to deploy a new model is doable, albeit often cumbersome. At best, it's just code changes. At worst, it involves retraining models. The crux of this for us is that it's especially cumbersome because we're in a regulated environment. Updating dependencies for already deployed models means lots of additional work and testing, and if there is model retraining involved, it's pretty much a non-starter. So, we'd much rather be able to have those older models continue to deploy the versions of the dependencies they were trained on, rather than forced updating. At this point, we're able to work around that without too much effort. That being said, we know it's not scalable for us in the long term, so we are going to have to find some workaround or alternative solution. I imagine that's not too different from what Sudeep has passed on. I appreciate you asking!

@Sean Sall I think the angle we struggle with is if this is actually going to make the whole environment more complicated. Having multiple dependency environments has a little bit of a code smell to it. It doesn't seem like a best practice. I understand the regulatory constraint and the annoyance of the use case in general. Is there any way to standardize the training environment dependencies? Or deploy across different services? Is the goal to be able to A/B test with a new model?

Appreciate the thoughtful response here @Tim Liu!

Having multiple dependency environments has a little bit of a code smell to it.

Fair, I can understand that. I think from my experience it's a fairly common problem to run into in a regulated environment. I'd love to avoid it, but that would mean continually devoting a lot of time to updating the dependencies of old models, and we'd love to not have to do that.

Is there any way to standardize the training environment dependencies?

We do already do this - any models trained today are on the same dependencies. The issue is that we can't retroactively standardize the environment for models that are already trained - we have some models that were trained ~2 years ago. With the way Bento works now (at least with our current setup), we're forced to update those models so that they are on the dependencies that we are currently training on.

Or deploy across different services?

This is what I'm trying to understand now, and with the above question. I'm not exactly sure what this looks like. With the limited understanding I have now I think the answer is yes, but it's much more cumbersome / painful than our current setup, and might cost us meaningfully in terms of throughput.

Is the goal to be able to A/B test with a new model?

No, I'm just talking about models that do different things. We have model X that we built ~2 years ago, and now we have model Y that we built 2 weeks ago (these times are arbitrary). Model Y is dependent on the output from Model X, and so deploying them within the same service is pretty beneficial / efficient. But, that means that we then have to get model X onto the same dependencies as model Y. That's a lot of extra work and has lots of implications for our ML folks as they develop (e.g. being pinned to certain dependencies).

Ah, that’s interesting I didn’t understand that last point. Let me take that feedback back to the team

@Tim Liu thank you, really appreciate it! We're certainly open to deploying to multiple services, but just trying to find the optimal path forward.

Sure, I’ll send you a dm