I'm seeing some interesting behaviour in my deployment of faster-whisper, logs here showing queue time (defined as time between bentoml svc beginning execution and runner beginning execution). I see that some requests get stuck and start to build up much more of a queue for some specific runners, there are some requests that ought to be sent to other runners that are less busy, so the requests wouldn't spend so long queueing.

api_v2-whisper-1  | 2024-04-17 14:02:55.996 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=1.8219. model_execution_time_s=1.3762, runner_queue_time_s=0.4457, runner_id=139997053363856, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:02:57.308 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=5.6077. model_execution_time_s=2.3055, runner_queue_time_s=3.3022, runner_id=139820294678096, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:02:58.174 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=2.4474. model_execution_time_s=0.5741, runner_queue_time_s=1.8734, runner_id=139820294678096, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:02:58.310 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=2.4084. model_execution_time_s=1.7065, runner_queue_time_s=0.7019, runner_id=140034448911888, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:02:58.738 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=1.6986. model_execution_time_s=0.7371, runner_queue_time_s=0.9615, runner_id=139820294678096, server_id='zSgj'
api_v2-whisper-1  | 2024-04-17 14:03:00.076 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=5.4660. model_execution_time_s=1.3389, runner_queue_time_s=4.1271, runner_id=139820294678096, server_id='zSgj'
api_v2-whisper-1  | 2024-04-17 14:03:00.550 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=1.4659. model_execution_time_s=0.6177, runner_queue_time_s=0.8482, runner_id=140034448911888, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:03:00.691 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=11.1913. model_execution_time_s=0.4736, runner_queue_time_s=10.7177, runner_id=139820294678096, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:03:00.923 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=3.0950. model_execution_time_s=0.7635, runner_queue_time_s=2.3315, runner_id=139820294678096, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:03:03.693 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=3.5236. model_execution_time_s=2.1864, runner_queue_time_s=1.3373, runner_id=139997053363856, server_id='oWJg'
api_v2-whisper-1  | 2024-04-17 14:03:05.903 | INFO     | service:transcribe:134 - Generated transcription in api_execution_time_s=5.8310. model_execution_time_s=5.7058, runner_queue_time_s=0.1251, runner_id=140034448911888, server_id='zSgj'

Here's my

configuration.yml

:

api_server:
  workers: 3
  traffic:
    timeout: 90
    max_concurrency:
  # max_runner_connections: 1
runners:
  resources:
    <http://nvidia.com/gpu|nvidia.com/gpu>: 1
  workers_per_resource: 3
  traffic:
    timeout: 60
    max_concurrency:

Does anyone have any thoughts on the cause of this issue, and any recommendations for solutions? I'm thinking about experimenting with the max_runner_concurrency or reducing the number of runners (taking a hit to throughput, in exchange for smoother latency across requests)