Hi! I've updated BentoML in my API from version 0...
# ask-for-help
x
Hi! I've updated BentoML in my API from version 0.13.1 to 1.2.17, and while performing load tests, I've detected a considerable memory increase (around 500MB) despite trying to keep the configuration as similar as possible to what we had before. Is this possible? What could be causing this memory increase? Thanks!
c
Could you share a bit more about your code and model? In most of our tests, 1.2+ versions should be more efficient
m
There was a memory leak issue solved in 1.2.19
👍 1
x
I load several custom models, but they are exactly the same as those loaded with the previous version. I define a service with all default settings except for the workers (I use the same number of workers as in the other version, 2). Within the service, I define 7 @api endpoints that load (the first time they are called) the different models and call them to get the prediction.
Hi, I have tested 1.3.0 and I see the same memory increase compare with 0.13.1 This is an example of the code I use to load the models and make the predictions:
Copy code
bentoml_workers = os.getenv('BENTOML_WORKERS', '2')

model_1_artifacts = {
    "model_1_feature_extraction": FeatureExtractorArtifact(
        "model_1_feature_extraction")}

model_2_artifacts = {
    "model_2_feature_extraction": FeatureExtractorArtifact(
        "model_2_feature_extraction")}

model_3_artifacts = {
    "model_3_feature_extraction": FeatureExtractorArtifact(
        "model_3_feature_extraction")}


@service(
    workers=int(bentoml_workers)
)
class Service():
    """
    Service to manage document models, requests and their predictions.
    """
    def __init__(self):
        self.model_1_class = None
        self.model_2_class = None
        self.model_3_class = None

    @api()
    def model_1(self, process_info: JSONType, ctx):
        """
        """
        try:
            headers = set_thread_log_variables(LOG_INFO, ctx)
            process_info["process_name"] = "model_1"
            if self.model_1_class is None:
                self.model_1_class = Model1Class(
                    model_1_artifacts)
            with StopWatch(process_info["process_name"]):
                response = self.model_1_class.model_1_response(
                    process_info)

        except Exception as exception:  # pylint: disable=broad-except
            response = error_handler(exception)
            ctx.response.status_code = response["error_code"]

        ctx.response.headers.update(headers)
        return response
    
    @api()
    def model_2(self, process_info: JSONType, ctx):
        """
        """
        try:
            headers = set_thread_log_variables(LOG_INFO, ctx)
            process_info["process_name"] = "model_2"
            if self.model_2_class is None:
                self.model_2_class = Model2Class(
                    model_2_artifacts)
            with StopWatch(process_info["process_name"]):
                response = self.model_2_class.model_2_response(
                    process_info)

        except Exception as exception:  # pylint: disable=broad-except
            response = error_handler(exception)
            ctx.response.status_code = response["error_code"]

        ctx.response.headers.update(headers)
        return response
    
    @api()
    def model_3(self, process_info: JSONType, ctx):
        """
        """
        try:
            headers = set_thread_log_variables(LOG_INFO, ctx)
            process_info["process_name"] = "model_1"
            if self.model_3_class is None:
                self.model_3_class = Model3Class(
                    model_3_artifacts)
            with StopWatch(process_info["process_name"]):
                response = self.model_3_class.model_3_response(
                    process_info)

        except Exception as exception:  # pylint: disable=broad-except
            response = error_handler(exception)
            ctx.response.status_code = response["error_code"]

        ctx.response.headers.update(headers)
        return response

class Model1Class():
    """
    """
    def __init__(self, artifacts):
        self.artifacts = artifacts
        load_artifacts(artifacts)

    def model_1_response(self, process_info):
        """
        """
        # Code to make the prediction

def load_artifacts(artifacts):
    """
    Function to load artifacts

    artifacts: dict
        dict that contains artifacts
    """
    for artifact in artifacts:
        artifacts[artifact].load(artifacts[artifact].original_filepath)

class FeatureExtractorArtifact(CustomArtifact):
    """
    """

    def __init__(
        self,
        name
    ):
        model_descriptor = ANTISPOOFING_MAP[name]
        self.original_filepath = model_descriptor.filepath
        self.feature_extractor_name = model_descriptor.feature_extractor_name
        self._custom_objects = model_descriptor.custom_objects if \
            model_descriptor.custom_objects else None
        self.feature_vector = None

    def check_input(self, *args):  # pylint:disable=no-self-use
        """"""
        return args

    def preprocess(self, *args):  # pylint:disable=no-self-use
        """"""
        return args

    def detect(self, image, process_info,  # pylint:disable=arguments-differ
               threshold, feature_extractor, do_calibration=True):
        """
        """
        # Code to make the prediction

class CustomArtifact():
    """
    """
    def load(self, path):
        """
        """
        if os.path.isfile(path):
            self._model = keras.models.load_model(  \
                # pylint: disable=attribute-defined-outside-init
                path,
                custom_objects=self._custom_objects,  # pylint: disable=no-member
                compile=False
            )
        elif os.path.isdir(path):
            self._model = keras.models.load_model(  \
                # pylint: disable=attribute-defined-outside-init
                self._model_file_path(path),  # pylint: disable=no-member
                custom_objects=self._custom_objects,  # pylint: disable=no-member
                compile=False
            )
That 500mb memory increase that is observed, could it be due to new dependencies that you use in the new versions compared to 0.13.1? I'm just checking because I increase the memory when I'm actually using the same models. Do you see any more efficient way to load and use the models that could help me not use so much memory? Thanks for the help