Hello I m trying to build a bentoml application with `Faster BentoML #ask-for-help

Hello! I'm trying to build a bentoml application w...

Anderson Karl

07/16/2024, 2:25 PM

Hello! I'm trying to build a bentoml application with

Faster-Whisper

for infference. But i'm getting this error when running the container: bentoml: 1.2.20 faster-whisper: 1.0.3 ctranslate2: 4.3.1

Copy code

2024-07-16T13:41:29+0000 [ERROR] [entry_service:AudoSia:1] Initializing service error
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/_bentoml_sdk/service/factory.py", line 251, in __call__
    instance = self.inner()
               ^^^^^^^^^^^^
  File "/home/bentoml/bento/src/service.py", line 22, in __init__
    self.model = WhisperModel(model_name_or_path, device="cuda", compute_type="float16")
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/faster_whisper/transcribe.py", line 145, in __init__
    self.model = ctranslate2.models.Whisper(
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: CUDA failed with error CUDA driver version is insufficient for CUDA runtime version

Anderson Karl

07/16/2024, 3:52 PM

I did manage to make it work! Setting on

bentofile.yaml

Copy code

docker:
  cuda_version: "12.1.1"
  distro: debian
  python_version: "3.11.7"

Also add on some specific Cuda requirements

requirements.txt

Copy code

nvidia-cublas-cu12 
nvidia-cudnn-cu12

And we can't forget to add the

--gpus all

docker run

command.

👍 1

Jian Shen Yap

07/17/2024, 9:07 AM

Faster-Whisper

a pytorch model ?

Anderson Karl

07/17/2024, 12:07 PM

It uses a CTranslate2 backend. So i need to convert the Transformers model to CTranslate2 backend if i want to use a specific model. It seemed to work well with WhisperX as both were integrated.

Open in Slack

Previous Next