This message was deleted.
# ask-for-helps
Slackbot
10/17/2022, 5:37 PMThis message was deleted.
c
Chaoyu
10/17/2022, 5:39 PMthat looks right! it should create two endpoints
/predict/predict_2🙏 1
s
Shihgian Lee
10/17/2022, 7:30 PM@Chaoyu Thanks! How do I verify if I setup adaptive batching correctly in the code?
c
Chaoyu
10/17/2022, 7:56 PM@Shihgian Lee do you have
batchable: True--productionChaoyu
10/17/2022, 7:57 PMwe recently added batching related metrics too, you can view batch size histogram from a grafana UI too https://github.com/bentoml/BentoML/pull/2902
Chaoyu
10/17/2022, 7:59 PMNote that batching only happens at the runner level now, not the API level, if you need to batch custom code/operations, you can do that via custom runners
s
Shihgian Lee
10/17/2022, 8:10 PMHi @Chaoyu Yes, I have batchable set to true in the runner. The adaptive batching for BentoML 1.0 is very confusing for me. I am still working it out with Sean. Can you take a look at this thread when you get a chance? Sean might be tied up with work and Kubeflow summit. This is the last step that is blocking me from deploying to Kubernetes cluster for load testing.
Shihgian Lee
10/17/2022, 10:15 PM(removed due to posting to the wrong thread)