This message was deleted BentoML #ask-for-help

This message was deleted.

Slackbot

01/20/2023, 12:43 AM

This message was deleted.

larme (shenyang)

01/20/2023, 5:29 AM

Shouldn’t be that slow. Is

torch.cuda.is_available()

still return

False

larme (shenyang)

01/20/2023, 5:30 AM

If you run

bentoml serve

, you should see vram usage in the output of

nvidia-smi

, or else the model is not utilizing GPU

latemetal

01/20/2023, 4:56 PM

its using cuda now

latemetal

01/20/2023, 4:56 PM

I fixed that

latemetal

01/20/2023, 4:56 PM

my gpu/vram is like 100%

latemetal

01/20/2023, 4:57 PM

i have to set

PYTORCH_CUDA_ALLOC_CONF=max_split_size_mb:768

or lower otherwise i get memory errors

5 Views