This message was deleted.

Hi Jack.

Because of this, each time they use it, it seems like the service has to restart and load all the runners and it takes a long time for them to be able to actually use the endpoint.

I don't think it is the default behavior of bentoml. This might be caused by a service.py not in best practice.

Hi @Jiang, sure thing, it’s possible I’ve made a mistake in the service.py. The service.py is essentially this:

# define IO descriptors
# ...
success_probability_input_descriptor = (JSON(pydantic_model=SuccessProbabilityInputDescriptor))
success_probability_output_descriptor = (JSON(pydantic_model=SuccessProbabilityOutputDescriptor))
# ...

# Load Bento Models from Google Cloud Storage; also convert them to Runners
bento_models, runners = load_bentos_from_gcs()

svc = bentoml.Service("probability-models-bento", runners=list(runners.values()))


def success_probability_pipeline(input_data):
    # do stuff with input_data


@svc.api(input=success_probability_input_descriptor,
         output=success_probability_output_descriptor,
         route="success/predict_proba")
def success_predict_proba(
        input_data: SuccessProbabilityInputDescriptor) -> SuccessProbabilityOutputDescriptor:
    validated_results = success_probability_pipeline(input_data)
    return validated_results

# other pipelines ...

has to restart and load all the runners

Can you tell me about the relevant phenomenon behind this? And would you like provide the source of

load_bentos_from_gcs

?

@Jiang Sure, this is `load_bentos_from_gcs`:

def load_bentos_from_gcs(bento_framework_model_names, load_from_gcs=True):
    bento_models = {}
    runners = {}
    for model_tmp in bento_framework_model_names:
        model_name_tmp = model_tmp["model_name"]
        framework = model_tmp["framework"]
        if load_from_gcs:
            <http://bentoml_logger.info|bentoml_logger.info>(f"Importing {BUCKET}/{model_name_tmp}.bentomodel from GCS")
            try:
                bentoml.models.import_model(f"{BUCKET}/{model_name_tmp}.bentomodel")
            except (BentoMLException, ImportServiceError) as e:
                bentoml_logger.warning(f"The Bento model is already in the store. No harm done, though! "
                                       f"The {e.__class__.__name__} exception caught from Bento was:"
                                       f"\n\n\t'{e}'\n\nProceeding with the rest of the build.")

        <http://bentoml_logger.info|bentoml_logger.info>(f"Loading in the {model_name_tmp}:latest model using the {framework} framework.")
        if framework == "sklearn":
            bento_model = bentoml.sklearn.get(f"{model_name_tmp}:latest")
        elif framework == "picklable_model":
            bento_model = bentoml.picklable_model.get(f"{model_name_tmp}:latest")

        <http://bentoml_logger.info|bentoml_logger.info>("Success.")
        <http://bentoml_logger.info|bentoml_logger.info>(f"Converting the model to a runner")
        runner = bento_model.to_runner()
        bento_models[model_name_tmp] = bento_model
        runners[model_name_tmp] = runner
    return bento_models, runners

Sure, I understand that: 1. You are using the Cloud Run Serverless service, which will automatically shut down the server after a certain period of inactivity (about 15 minutes) to save costs. 2. Your Bento model is being re-downloaded every time the server starts. The combination of these two factors is causing the current issue of unavailability.

@Jiang, thank you very much for these options and taking the time to explain! Yes, I think our immediate solution will be setting Cloud Run to always be on. We chose not to import the model during image build because we liked the idea of keeping the models separate from the service. Our reasoning was that if we were to retrain a model, we could simply just upload the new model to GCS and restart the service. With a retrained model using the image build approach, we would have to rebuild the image and redeploy. Maybe that doesn’t actually save that much down time, though. I will look into BentoCloud again. Is it an easy transition from Cloud Run to BentoCloud?

It seem that we need a new revision name?