Hi everyone, I have migrated onnx bentoml service ...
# ask-for-help
m
Hi everyone, I have migrated onnx bentoml service from v0.12.0 to v.1.0.7. and found that when
batchable=False
performance with old bentoml release was better than with new release (better latency/throughput and no failures). ONNX models didn't have changed. Please see attached the 2 locust comparisons with exact same conditions. What could explain this?
Would anyone have an idea?