This message was deleted.

This looks like there is a memory leak in your service. I am running bentoml 1.0.10 on kubernetes pod and the memory usage has been pretty constant (straight line).

i’m on 1.0.13 and i didn’t do anything fancy with my service.py

import os

import bentoml
from <http://bentoml.io|bentoml.io> import JSON

model_uri = os.getenv("MODEL_URI")

my_runner = bentoml.mlflow.import_model(f"my_model", model_uri).to_runner()

svc = bentoml.Service("my_service", runners=[my_runner])


@svc.api(input=JSON(), output=JSON())
def predict(input_arr):
    return my_runner.predict.run(input_arr)

and I started it in a Dockerfile like this ($BASE_IMAGE just provide python packages)

ARG BASE_IMAGE
FROM --platform=linux/amd64 $BASE_IMAGE

WORKDIR /
COPY service.py service.py

EXPOSE 3000:3000

ENTRYPOINT ["bentoml", "serve", "service:svc", "--reload"]

not sure what I did wrong here

Yeah, that looks pretty straightforward to me. If I encountered this problem, I would try the following: 1. Deploy an iris classifier bento from the tutorial without any customization to find out if the memory issue is still there. 2. Use a profiler tool like memray to profile your service locally.

i realized that it’s the flag

--reload

that causes this mem usage… after i changed it to

--production

it’s a stable flatline now

@James Zhang That is good to hear! Just curious, what do you use to monitor BentoML 1.0.x? I use Datadog. After the upgrade from 0.13 to 1.0.x, I lost visibility to the code in the runner; I don't see the metrics for external calls to Google Bigquery, etc. Please see my recent conversation here if you are curious.

we are using basic k8s prometheus metrics, so i just watch the RAM usage of the pod, nothing fancy 🙂

Thank you for the feedback!

thanks for your info! i did try to use

containerize

but then i found out that in our current infrastructure it’s not possible because our EKS doesn’t support dind anymore… i also tried to use

podman

buildah

as the

backend

argument, they didn’t work either because i could only install older versions of them but

containerize

seems to invoke methods of the newer versions of them, after some tries i gave it up…

Hi @James Zhang You don't have to use containerize. I am not using it. I am using our DevOps script to containerize the Dockerfile. I am using Dockerfile template to integrate our datadog tracing with BentoML. The

bentoml build

command outputs my customized Dockerfile. Then I just point my containerization script to the generated custom Dockerfile. If your current approach is simple and works for you, that is great! No need to use the Dockerfile template.