I am facing while deploying Depth-Anything-V2-Small-hf model from transformers. i was trying to run the model on mps backend since i am using an apple silicon. Here is my code :

Download_model.py

import bentoml
import torch
from transformers import pipeline

# depth_pipeline = pipeline(task="depth-estimation", model="LiheYoung/depth-anything-large-hf")
device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")

depth_pipeline = pipeline(task="depth-estimation", model="pcuenq/Depth-Anything-V2-Small-hf", device=device)
depth_pipeline.model.to(device)
bentoml.transformers.save_model("depth_pipeline_midasmal", depth_pipeline)

service.py

import bentoml
import torch
from bentoml.io import NumpyNdarray, JSON
import numpy as np

# pipeline_tag = "depth_pipeline:4txpx2bunw7rauqk"

pipeline_tag = "depth_pipeline_midasmal:ii7p42rutovfiuqk"

device = torch.device("mps" if torch.backends.mps.is_available() else "cpu")


pipe_runner = bentoml.transformers.get(pipeline_tag).to_runner()

depth_service = bentoml.Service("depth_estimation_service", runners=[pipe_runner])

@depth_service.api(input=JSON(), output=NumpyNdarray())
async def estimate(json_data):
    image_np = np.array(json_data["data"])
    if len(image_np.shape) == 2:
        image_np = image_np[..., np.newaxis]
    depth_map = await pipe_runner.async_run(image_np)
    return depth_map

test_endpoint.py

import requests
import numpy as np
import json
import cv2

image_path = '/Users/hasibulhasan/Downloads/PXL_20240607_113141785.jpg'

image = cv2.imread(image_path)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
image_np = np.array(image, dtype=np.float32)

image_json = {"data": image_np.tolist()}

url = "<http://localhost:3000/estimate>"

# headers = {'Content-Type': 'application/x-npy'}
response = requests.post(url, json= image_json)

if response.status_code == 200:
    print("hoise")
    depth_map = np.frombuffer(response.content, dtype=np.float32)
    depth_map = depth_map.reshape(image_np.shape[:2])  # Reshape it back to the image dimensions
    print("Received depth map with shape:", depth_map.shape)
else:
    print("Request failed with status code:", response.status_code)
    print("Response:", response.text)

server log

(bento) (base) [~/github/bento]$ bentoml serve --timeout 1000
2024-06-27T21:42:53+0600 [INFO] [cli] Environ for worker 0: set CPU thread count to 8
2024-06-27T21:42:53+0600 [INFO] [cli] Prometheus metrics for HTTP BentoServer from "." can be accessed at <http://localhost:3000/metrics>.
2024-06-27T21:42:55+0600 [INFO] [cli] Starting production HTTP BentoServer from "." listening on <http://0.0.0.0:3000> (Press CTRL+C to quit)
2024-06-27T21:42:59+0600 [INFO] [api_server:depth_estimation_service:8] 127.0.0.1:58097 (scheme=http,method=GET,path=/,type=,length=) (status=200,type=text/html; charset=utf-8,length=2945) 0.411ms (trace=75780dd9f16b36510197c1fe6ed99b73,span=5f33fb470fbe8985,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:42:59+0600 [INFO] [api_server:depth_estimation_service:8] 127.0.0.1:58097 (scheme=http,method=GET,path=/static_content/swagger-ui.css,type=,length=) (status=200,type=text/css; charset=utf-8,length=152059) 16.204ms (trace=095c7191252e02525684be8f25114fd4,span=20650d31ac5e04a4,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:1] 127.0.0.1:58101 (scheme=http,method=GET,path=/static_content/swagger-initializer.js,type=,length=) (status=200,type=application/javascript,length=331) 9.405ms (trace=f1c06f1e904710b79dac4dd785e84b08,span=7f388fe9f2881dd3,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:7] 127.0.0.1:58100 (scheme=http,method=GET,path=/static_content/index.css,type=,length=) (status=200,type=text/css; charset=utf-8,length=1127) 2.234ms (trace=9acc04d5b085b108bbdb0acc6aa7fc19,span=48aa0d612ab16cfe,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:7] 127.0.0.1:58099 (scheme=http,method=GET,path=/static_content/swagger-ui-standalone-preset.js,type=,length=) (status=200,type=application/javascript,length=230777) 30.571ms (trace=4b9e66d72fb932e72a8ef0feffddcf9e,span=8510f7ae81355035,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:1] 127.0.0.1:58098 (scheme=http,method=GET,path=/static_content/swagger-ui-bundle.js,type=,length=) (status=200,type=application/javascript,length=1415333) 47.594ms (trace=7e71cc6c2f94320aae1f5c87fa3405f4,span=aef0aa30ff6028eb,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:1] 127.0.0.1:58098 (scheme=http,method=GET,path=/docs.json,type=,length=) (status=200,type=application/json,length=4706) 13.539ms (trace=11df5d583575ff81fe3587fc6f610ef8,span=58f0c0f2f323a921,sampled=0,service.name=depth_estimation_service)
2024-06-27T21:43:00+0600 [INFO] [api_server:depth_estimation_service:7] 127.0.0.1:58099 (scheme=http,method=GET,path=/static_content/favicon-dark-32x32.png,type=,length=) (status=200,type=image/png,length=654) 1.908ms (trace=34d1d44fa8be3432149daa269d073c1c,span=70098bea9a02f4c1,sampled=0,service.name=depth_estimation_service)
/Users/hasibulhasan/miniforge3/envs/bento/bin/python /Users/hasibulhasan/github/bento/test_endpoint.py
2024-06-27T22:00:08+0600 [ERROR] [api_server:depth_estimation_service:7] Exception on /estimate [POST] (trace=942730e7d209e91eeab4df4d1480d854,span=7ab027b0153497ee,sampled=0,service.name=depth_estimation_service)

I am not getting any response even if i set the timeout to 1000s . It is taking around 4 to 5 gb memory .

By default the bentoml server runs N workers, where N is equal to cpu cores. Model is ~ 1.34GB so probably you have 4 cpu workers. Did you try running the plain inference code ? How much time does it take ?