Hello everyone, I have a BentoML service configured for maximum concurrency of 4 on my local machine. However, the incoming requests seem to be getting queued instead of being processed in parallel as expected. What could be the reason behind this behavior?

maximum concurrency determines how many request a service should be processing at once, it is unlikely to affect parallel processing. Can you share your service code so we can go through it and see where is the bottleneck

hi, @Jian Shen Yap This is my service.py

import bentoml
import os
from typing import Any
from LargeTranslate import Transcribe
from pathlib import Path
import shutil
import torch
import time

@bentoml.service(

http = {
        "cors" : {
        "enabled": True,
        "access_control_allow_origins": "*",
        "access_control_allow_methods": ["GET", "OPTIONS", "POST", "HEAD", "PUT"],
        "access_control_allow_credentials": True,
        "access_control_allow_headers": ["*"],
        "access_control_allow_origin_regex": "https://.*\.my_org\.com",
        "access_control_max_age": 1200,
        "access_control_expose_headers": ["Content-Length"]
    }
    },
    traffic={
        "timeout": 600,
        "max_concurrency": 3,

    },
    workers = 3,

)
class AsrBentoGPU:
    def __init__(self):
        self.model = Transcribe()

    @bentoml.api
    async def transcribe(self, form_data: Path) -> Any:

        print("========REQUEST RECEIVED==========")
        script_dir = Path(__file__).resolve().parent
        # audio_path = os.path.join(script_dir, "uploads")
        actual_path = form_data.absolute()
        shutil.move(actual_path,script_dir)
        audio_final_path = os.path.join(script_dir, form_data.name)

        print(100*"*")
        print(f"===============cuda is available===================: {torch.cuda.is_available()}")
        print(100*"*")
        print(f"======================Audio_path===================:{audio_final_path}")
        print(100 * "*")

        t1 = time.time()
        transcription = await self.model.run(audio_final_path)
        t2 = time.time()

        print(100*"*")
        print(f"============Transcription:===================== {transcription}")

        print(100*"*")
        print(f"==================Inference_time======================: {t2-t1}")
        print(100*"*")

        if os.path.exists(audio_final_path):
            os.remove(audio_final_path)
            print(f"====================Deleted file=======================: {audio_final_path}")
        return transcription

It looks okay, what are your expected behaviour and what is happening? Could you elaborate more on that

What's my expectations is that when I send 2 requests at a time (without any delay) the requests should process parallelly , but after completion of one request the 2nd request is accepted by the API currently

@Akash can you confirm when you do

bentoml serve

, 3 workers is being spun up?

The workers are spawned, but if I keep worker to 1, I am facing the above issue

That is expected with 1 worker, that you can only process 1 request at a time due to how python works

Okay, but the same thing isn't the same if I use flask as my framework right?