This message was deleted.
# announcementss
Slackbot
12/04/2020, 9:11 AMThis message was deleted.
c
Chaoyu
12/04/2020, 9:17 AMBentoML has its own async layer built with aiohttp which is optimized for ML model serving
Chaoyu
12/04/2020, 9:18 AMThe flask part is the backend model server, that we could replace with something else later on and does not affect the overall performance that much
Chaoyu
12/04/2020, 9:18 AMThere’s a benchmark project under the bentoml GitHub org page
w
Wang
12/04/2020, 9:40 AMyeah I've checked the bentoml/benchmark before, and I'm just thinking maybe a contrast matrix or something like that might make it more attractive. bentoml has made model to service much simpler, I like the idea of packing model with the service code and generating docker image, but some users could hesitate on performance, an intuitive show-out might do some help.
👍🏼 1
Wang
12/04/2020, 9:47 AMby the way, last few days I'm working on how to make my transformers model run on bentoml with gpu, I solve that problem with a customize bento artifact(found some sample code on slack), but I'm wondering if there is an easier way like changing some setting in bento service?
Wang
12/04/2020, 10:06 AM Copy code
# now i do like this
class MyTMA(TransformersModelArtifact):
def pack(self, model):
super().pack(model)
self._model["model"].cuda()
@bentoml.env(infer_pip_packages=True)
@bentoml.artifacts([MyTMA("albert")])
class TransformerService(bentoml.BentoService):
@bentoml.api(input=JsonInput(), batch=False)
def predict(self, parsed_json):
passj
Jian Shen Yap
12/04/2020, 3:40 PMoptimized for ML model servingj
Jiang
12/05/2020, 3:00 AM@Jian Shen Yap Micro-batching is the major approach. We also made other optimizations like Dataframe merging (calling
pandas.read_jsonJiang
12/16/2020, 9:38 AMYeah, appling
fastAPIsanic👍 3
Jiang
12/16/2020, 9:45 AMThe exception here is the batching server. It holds thousands of connections from clients, merges inputs to batch, and dispatches to model servers. They are IO-bound. That’s why it's implemented with asyncio. In addition to https://github.com/bentoml/benchmark, there is a user in the channel who just made some benchmarks comparing with the tensorflow-serving. Bentoml and tf-serving have similar latency and throughput under high concurrent requests.
j
Jian Shen Yap
12/17/2020, 3:38 AM@Jiang if you are referring to me, yes, I did a benchmark comparing with tf-serving. on low load, few request at a time, tf serving will be able to achieve very low latency. I supposed this is due to tf serving not needing to have python overhead. but at high load thousands RPS, bentoML are able to match the throughput and beat the latency of tf serving. for some reason the response time for bentoML is very stable, while tfserving has higher variation in response time (difference between p5, p50 , p95 response time is huge)
🤣 1
j
Jackson Hsieh
01/14/2021, 3:16 AMhi @Jiang, I compare bert with bentoml and tf-serving. And I get a huge gap of the performance. Betoml will get lots of 429 response and slow responses while tf-serving can success fast and completely. Do you have any ideas how to improve the service?
j
Jiang
01/14/2021, 5:06 AM@Jackson Hsieh I also tried BERT with bentoml. There are two possible reasons for your problem:
• Can you find many lines like
triggered tf.function retracing@tf.function(experimental_relax_shapes=True)j
Jackson Hsieh
01/14/2021, 7:28 AM@Jiang I test my service and find most of the micro-batch will have size 1; only few times will have size larger one. Do I need to tune mb_max_latency or mb_max_batch_size ?
Jackson Hsieh
01/14/2021, 10:05 AMAlso, when I want to increase our RPS but I will got lots of 429. Do I need to increase the mb_max_latency?