BentoML

Hi. An occasional 503 error is a normal pressure release mechanism. May I ask if it occurs consistently or only at the beginning of the service launch?

I would have thought 429 would be a more conventional back-off status code?

But my question was really around why it occurs in production mode, but in debug mode it can process a higher throughput without any errors

In some further investigation I found that it relates to the adaptive batching - it I disable that the throughput drops but I don't get errors