This message was deleted.
# ask-for-helps
Slackbot
11/04/2022, 9:37 PMThis message was deleted.
b
Bo
11/07/2022, 4:49 PMGive the new bentoml client a try. It should works for you this front.
On init_local. This one doesn’t have batching enabled, it’s slower
m
Mike Kuhlen
11/07/2022, 5:25 PMBy “new bentoml client”, do you mean the
Client.from_urlMike Kuhlen
11/07/2022, 5:25 PMThe “broken pipe” issue makes it impossible for me to run large batch requests over the network API. That’s one of the reasons I’ve been falling back to “local eval”.
c
Chaoyu
11/07/2022, 10:47 PMSorry about the delay @Mike Kuhlen The warning is only made for online server and it should be ok for offline batch scoring. It will be equivalent to a regular python function call and occupy memory in the same python process.
gratitude thank you 1
Chaoyu
11/07/2022, 10:48 PMOur team has been investigating the “broken pipe” issue for the past week, a fix for it is coming out soon, we expect to have it in the next release
m
Mike Kuhlen
11/07/2022, 10:48 PMThat it awesome! It sounded like a real tricky one.
c
Chaoyu
11/07/2022, 10:50 PMyea, it had to do with some fundamental python networking library. Hopefully, we will switch all that to Rust/C++ next year 😜
🧐 2
Chaoyu
11/07/2022, 10:51 PMwe are also building more batch inference features that will take advantage of multi CPU cores or on a Spark/Dask cluster. It will also provide lower-level APIs for achieving what you were looking to implement here.
m
Mike Kuhlen
11/07/2022, 10:52 PMLooking forward to it. Lemme know if you need someone to beta test that.
3 Views