https://bentoml.com logo
Join Slack
Powered by
Am I setting up my service correctly? In `service....
# ask-for-help
j
Am I setting up my service correctly? In 
service.py
I create my service by passing in my ~15 runners. Before I pass the runners in, I first load them into memory. But what I’m seeing is that the runners are being loaded in something like 50+ times each. Because each runner is large, I get memory issues right away. Has anyone seen this issue before? Is there something upstream that runs 
service.py
in a loop perhaps?
j
can you please share your code so that we can understand what you are trying to do!
j
This is the crux of it @Jian Shen Yap ; this is `service.py`:
Copy code
import bentoml
import pandas as pd

from constants import BUCKET


class Pipeline1:
    def __init__(self, bento_models, runners):
        self.bento_models = bento_models
        self.runners = runners
        self.model_name = 'model1'
        self.custom_objects = bento_models[self.model_name].custom_objects
        self.nums = self.custom_objects['nums']
        self.cats = self.custom_objects['cats']
        self.encoder = self.custom_objects['encoder']
        self.model_version = self.custom_objects['model_version']
        self.feature_names = list(self.custom_objects['feature_names'])
        self.classes = ['negative', 'positive']

    def prepare_data(self, input_data):
        input_data = input_data[self.nums + self.cats]
        return self.custom_objects['prepare_data'](input_data, self.cats, self.nums, '.', self.encoder)

    def predict_proba(self, input_data):
        data = self.prepare_data(input_data)
        results = self.runners[self.model_name].predict_proba.run(data)
        results_df = pd.DataFrame(results, columns=self.classes)
        return results_df


def load_bentos_from_gcs(bento_framework_model_names, load_from_gcs=True):
    bento_models = {}
    runners = {}
    for model_tmp in bento_framework_model_names:
        model_name_tmp = model_tmp["model_name"]
        framework = model_tmp["framework"]
        if load_from_gcs:
            bentoml.models.import_model(f"{BUCKET}/{model_name_tmp}.bentomodel")
        if framework == "sklearn":
            bento_model = bentoml.sklearn.get(f"{model_name_tmp}:latest")
        elif framework == "picklable_model":
            bento_model = bentoml.picklable_model.get(f"{model_name_tmp}:latest")
        else:
            pass  # handle other cases
        runner = bento_model.to_runner()
        bento_models[model_name_tmp] = bento_model
        runners[model_name_tmp] = runner
    return bento_models, runners



bento_models, runners = load_bentos_from_gcs(BENTO_FRAMEWORK_MODEL_NAMES, load_from_gcs=False)
svc = bentoml.Service("probability-models-bento", runners=list(runners.values()))


pipeline1 = Pipeline1(bento_models, runners)


@svc.api(input=input_descriptor, output=output_descriptor, route=f"{pipeline1.model_name}/predict_proba")
def pipeline1_predict_proba(input_data: InputDescriptor) -> OutputDescriptor:
    validated_results = pipeline1.predict_proba(input_data)
    return validated_results
load_bentos_from_gcs()
is being called hundreds of times which means that the Bento models are being loaded in hundreds of times
len(BENTO_FRAMEWORK_MODEL_NAMES)
is 14
I’m on BentoML version 1.0.17 but I can’t afford to upgrade at this time
The loading of the models so many times seems like something is going wrong. This spikes the memory usage and crashes the service. Do you see anything I’m doing naively wrong? Let me know if you’d like more information. It might also be easier over a brief video call if you’re up for it. Thanks in advance @Jian Shen Yap
(Naturally there are 14+ pipeline classes; this is just a snippet of the code)
The number of times 
load_bentos_from_gcs()
is I think something like 
api_workers x num_runners x num_runners + num_runners
j
Copy code
pipeline1 = Pipeline1(bento_models, runners)
^ i think it comes from here. each Pipeline object contains all runner and is being referenced directly into @svc which creates a process for each api workers, which explain why you have a surge of memory.
plus in this way you are actually not using the runner architecture properly
I don't see how you are calling your 15 runners from your API server. maybe you could explain your use case here
j
The api function (
pipeline1_predict_proba
) calls 
pipeline1.predict_proba(input_data)
which uses the runner 
self.runners[self.model_name].predict_proba.run(data)
I have 14+ APIs that follow the same pattern
There are a couple APIs that need to use more than one runner. I agree that I shouldn’t probably pass in the full list of bento models or runners (though I thought it would be doing this by reference), but is it improper to pass in a runner like I am doing? Did I answer your question about my use case?
@Jian Shen Yap let me know if you have any thoughts or if I can provide more information, 🙏 thank you!
@Jian Shen Yap I updated my code so that each pipeline only gets the model and runner it needs instead of the entire list:
Copy code
bento_models, runners = load_bentos_from_gcs(BENTO_FRAMEWORK_MODEL_NAMES, load_from_gcs=False)
svc = bentoml.Service("probability-models-bento", runners=list(runners.values()))

name = "mymodel"
model = bento_models[name]
runner = runners[name]

pipeline1 = Pipeline1(model, runner)

@svc.api(input=input_descriptor, output=output_descriptor, route=f"{pipeline1.model_name}/predict_proba")
def pipeline1_predict_proba(input_data: InputDescriptor) -> OutputDescriptor:
    validated_results = pipeline1.predict_proba(input_data)
    return validated_results
I also commented out all other 
@svc.api
endpoints so that there’s just this one. And I’m still getting the huge surge of memory. Is this because I’m passing all the runners into the Service? How else do you recommend I do it? The runners are not that big of objects, ranging from 50 MB to 1.5 GB.
@Jian Shen Yap I have further simplified my code and updated it to test what’s going on:
Copy code
# service.py

bento_models, runners = load_bento_models(BENTO_FRAMEWORK_MODEL_NAMES, from_gcs=False)  # return (14 models, 14 runners)

name = "pipeline1"
model = bento_models[name]
runner = runners[name]
svc = bentoml.Service("probability-models-bento", runners=[runner])  # only pass in one runner, for testing purposes

@svc.api(input=input_descriptor, output=output_descriptor, route=f"{pipeline1.model_name}/predict_proba")
def pipeline1_predict_proba(input_data: InputDescriptor) -> OutputDescriptor:
    results = runner.predict_proba.run(input_data)
	return results
Now what I’m seeing is that 1. all 14 runners are loaded for each api worker 2. all 14 runners are loaded for each of the runners (of which there is only one in this simplified example) that is passed into the service. #2 would suggest that service.py gets run for each runner passed into Service(). This would be why when I pass in the list of all runners into Service(), I get more than 14*14 model/runner loads and the memory explodes. How do I load my models and runners but only the ones that are relevant to each runner?
And in case it’s helpful, all that the 
load_bento_models
function does is it loads all 14 of the models from the BentoML store using 
bentoml.sklearn.get(f"{model_name_tmp}:latest")
and then it uses 
.to_runner()
to convert to a runner.
@Jian Shen Yap I can simplify my question by asking this instead: • What (I think) I need: to call 
load_bento_models
in order to load in the models from the Bentoml store. • Observation: It seems as though 
service.py
gets run for 
cli
, each api server, and for each runner. • Observation: Because 
service.py
gets run for each runner, and each run loads in all models from the Bentoml store, I end up loading the models more than 
14^2
times. • QuestIon: How do I load the models in from the Bentoml store so that I can pass them into Service() without having this 
n^2
behavior?
j
Thanks for the detailed explanation. I'm still not understanding how would you like to run your 14 runners? do you expect 14 endpoints?
j
Yes. I have created 14 endpoints each of which uses just one runner. For this simplified example above, I’m just showing one endpoint defined, but in my actual code, I have 14.
j
Side question: would you consider using our 1.2 API?
j
We need to continue to support Python 3.7 at the moment and I believe 1.2 requires >= 3.8 right? I’ve looked into it. I would like to upgrade, but I’m hoping it can wait.
j
Now what I’m seeing is that
1. all 14 runners are loaded for each api worker
This is not the case. Runners by default are only loaded once. Here's an example
Copy code
# %%
# ########################################
# 🍀🍀🍀🍀🍀🍀🍀🍀 Vanilla Text Runner
# ########################################
import sys

import numpy as np

import bentoml


class TextRunnable(bentoml.Runnable):
    SUPPORTED_RESOURCES = ("cpu",)
    SUPPORTS_CPU_MULTI_THREADING = True

    def __init__(self):
        print("runner")

    @bentoml.Runnable.method()
    def generate(self, size_in_bytes: int) -> np.ndarray:
        # generate a payload of varying size of ndarray
        payload = np.random.rand(1, 1, int(size_in_bytes[0]))
        print(f"Size of payload = {sys.getsizeof(payload)}")
        return payload

text_runner = bentoml.Runner(TextRunnable)

# sse_runner = bentoml.Runner(SSERunnable)
svc = bentoml.Service("text_runner", runners=[text_runner])

@svc.api(input=bentoml.io.Text(), output=bentoml.io.NumpyNdarray())
async def count(size_in_bytes:str) -> str:
    size = int(size_in_bytes)
    ret = await text_runner.async_run(size_in_bytes*size)
    return ret
my CPU has 24 core, so it spins up 24 api workers but only 1 runner
your issue is that you are calling 
load_bento_models
in service.py so it loads all the model for each api worker, not the runner
j
Is your code above 
service.py
?
If you created 14 text_runners, how would that be different from what my 
load_bento_models
function is doing?
j
yes it's 
service.py
Another example
Copy code
# %%
########################################
# 🤖🤖🤖🤖🤖🤖🤖🤖 Transformer Service
########################################
import bentoml

print("hello")

summarizer_runner = bentoml.models.get("summarization:4r5m6nsrnwecfbo7").to_runner()

svc = bentoml.Service(
    name="summarization", runners=[summarizer_runner]
)

@svc.api(input=bentoml.io.Text(), output=bentoml.io.Text())
async def summarize(text: str) -> str:
    generated = await summarizer_runner.async_run(text, max_length=3000)
    return generated[0]["summary_text"]
you will see 
hello
printed many times, but there will only be 1 runner process if you do 
ps aux | grep python
i haven't figured out where your code has went wrong but perhaps this could give you some clue
j
This is because you only are loading in one model
j
onloading model into memory only happens within the Runner itself
j
So if you had
Copy code
summarizer_runner1 = bentoml.models.get("summarization:asdfasdf1").to_runner()
summarizer_runner2 = bentoml.models.get("summarization:asdfasdf2").to_runner()
...

svc = bentoml.Service(
    name="summarization", runners=[summarizer_runner1, summarizer_runner2]
)
Then I think you would see that each of 
summarizer_runner{n}
would get loaded in for each of the runners you pass into 
Service()
Resulting in getting loaded in 
14^2
times
j
Yes, but the actual loading of the model into memory only happens once in the Runner's Init https://github.com/bentoml/BentoML/blob/main/src/bentoml/_internal/frameworks/sklearn.py#L181 
bentoml.models.get("summarization:asdfasdf1").to_runner()
This does not start the runner itself but merely setting up the metadata etc.
I dropped a print statement into the 
Runnable
init
j
Sorry, to be clear, you dropped the print statement into the 
Runnable
init, which gets run at what point? 
.to_runner()
?
j
nope. 
.to_runner()
-> gives a 
Runner
Runnable
is an abstraction that represent the runner process that is spun up. this is an internal bentoml abstraction
converting to 
Runnable
happens somewhere when we call 
bentoml serve
j
Okay. So then is it safe to say that my call to 
load_bento_models
shouldn’t be problematic because it’s just calling 
bentoml.models.get
and 
.to_runner()
14 times for each of the 14 runners passed into 
Service()
, but the 
serve
should have it actually be loaded into memory just once (each)?
If not, then I don’t see how the functionality of my 
load_bento_models
is any different from the setup you have in your 
service.py
j
yes
you can try running your code
Copy code
bento_models, runners = load_bento_models(BENTO_FRAMEWORK_MODEL_NAMES, from_gcs=False)  # return (14 models, 14 runners)

name = "pipeline1"
model = bento_models[name]
runner = runners[name]
svc = bentoml.Service("probability-models-bento", runners=[runner])  # only pass in one runner, for testing purposes
in a python shell
multiple times
to see if it builds up memory usage
j
Thanks for the idea I tried this. When I load this code just once, the MiB Mem used (found with 
top
) goes from ~4030 to ~4661. When I run it in a for loop and run it 14*14 times, it goes from ~3987 MiB Mem used to ~6223 MiB Mem used. So it does build up memory usage but you’re right that it’s not as much as 14*14*(4661 - 4030)
If that’s the behavior for the runners, is it also the behaviors for the models? I need to load the models in because I need to access the 
custom_objects
when I’m defining my endpoint logic. Are the model objects also only loaded into memory just once or would they be loaded into memory 14*14 times?
j
if you are using the prebuilt runner, as you can see in the code, its loaded into the runner only when it starts up, i.e. 14 times
unless you somehow started 14*14 runner process
j
I must be starting a ton of processes? This shows my memory usage when running 
docker run -it --rm -p 3000:3000 probability-models-bento:vbeq37x47oc4mqqb serve --production --api-workers 2
which uses more a lot more memory than when I open up a Python shell and load in the 
Service
many, many times. This is at about 28GiBs memory used and counting, when I start to get “Cannot allocate memory” errors. Do you have any other ideas?
6 Views
Open in Slack
PreviousNext
Powered by