This message was deleted.

like how would we know if the gpu is running out of memory?

are you able to have any inference, or 503 only happens when you are sending alot of request? for gpu, you can perhaps use

nvidia-smi

to check on the gpu memory usagE?

is there no way to get the error that bentoml is seeing before returning Service Busy?

could you try to run

bentoml serve --debug

and see if you could find some logs there?

I just see stuff like this

bentoml._internal.marshal.dispatcher - Dynamic batching cork released, batch size: 3

503 typically happens when server is overloaded. Could you monitor the memory usage of the runner? @Aaron Pham do we know what are the other possible cases that might trigger 503 from the runner server?

This might happen when the dispatcher is overloaded, and it will reject incoming requests

Will increase the API_SERVER_TRAFFIC_TIMEOUT to 300s from 60s, but you can see from the logs that the 503 errors are barely 1s. The cpu usage is like really small (less than 1 core) and memory usage is constant at around 10GB but we have plenty more. Like the pods aren’t running out of memory and restarting or anything.

i think what aaron meant was to increase the

max_latency

for the batching configuration, what is your current settings for batching?

The

runner.traffic.timeout

is set to 300s and the

RUNNERS_BATCHING_MAX_LATENCY_MS

is 500ms

max_latency is the upper limit of latency on the runner. if you set it to 500ms and the runner' dispatcher predicts that a new incoming request will violate the limit, it will reject the request directly. So you should increase this to a limit that you are able to accept, ideally around the 80-90% of your application's SLA

Ohh, okay gotcha. Just to confirm it’s this right?

runners.batching.max_latency

(link to entry in default config) I guess I was confused because this is namespaced under the

batching

key so I thought it has to be related to batching. If batching was disabled, you’d still use this

runners.batching.max_latency

as the upper limit of the runner?

yes! you are right for both. if batching is disabled, the dispatcher will just forward the request as it is

Thank you everyone! Seems like increasing the max_latency timeout fixed the 503 error issue. I’ll keep an eye on the logs if we get some more.