This message was deleted.
# ask-for-helps
Slackbot
11/09/2022, 10:02 AMThis message was deleted.
e
Eric Riddoch
11/10/2022, 6:45 PMI think this may fall into the category of "offline" inference. I've heard that bento is working on a new offline inference capability, but I don't think it's out yet
f
Frédéric Kaczynski
11/14/2022, 12:13 PMThanks!
I think it's not properly "offline" inference. "Offline" inference means inference not in a real-time context (not in a user-facing app), like when you want to debug or during a batch job. In my case, the worker would be deployed in a real-time context, it's just that it would receive samples to predict from a queue instead of HTTP request.
I'll make an issue on the GitHub, I think it could be a nice feature to have 🙂
e
Eric Riddoch
11/14/2022, 9:09 PMSure. I think I'd seen bento use that term to describe: any time you run inference not in REST API mode.
Eric Riddoch
11/14/2022, 9:10 PMHere are the docs of the previous version of Bento I'd found on that: https://docs.bentoml.org/en/0.13-lts/guides/batch_serving.html
Eric Riddoch
11/14/2022, 9:11 PMThis seems potentially useful, but you have to invoke bento as a CLI rather than somehow call it directly from code which is disappointing to me. I'm hoping hte new version in bento 1.0.0+ allows for direct API calls in code
b
Bo
11/14/2022, 10:08 PM@Eric Riddoch With the recently released bentoml client, you should able to call directly in python environment
e
Eric Riddoch
11/15/2022, 12:32 AMIs it out?? Do you have docs I can read on it? Super excited
b
Bo
11/15/2022, 12:54 AMHere you go, fresh out of the printer(1.0.10) https://docs.bentoml.org/en/latest/guides/client.html
e
Eric Riddoch
11/16/2022, 6:52 AMNice! Is this the recommended approach for batch inference now? If so, I'd love to experiment with it. We use Metaflow for our batch jobs. If we can invoke this on large batches of data in our Metaflow DAGs... Bento will probably end up everywhere we run inference.
10 Views