Hi everyone, We successfully deployed our Bento container in our k8s environment and the container is running fine inside the pod. However, every time, a callout to the api endpoint is done, the pod re-runs its init, therefore stopping the service and not deliver a result. Based on debugging attempts I was able to identify the issue to be in the following piece of code:

outputs = model.generate(
            pixel_values.to(device),
            decoder_input_ids=decoder_input_ids.to(device),
            max_length=model.decoder.config.max_position_embeddings,
            pad_token_id=processor.tokenizer.pad_token_id,
            eos_token_id=processor.tokenizer.eos_token_id,
            use_cache=True,
            bad_words_ids=[[processor.tokenizer.unk_token_id]],
            return_dict_in_generate=True,
        )

with the following config for model, processor and device:

processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
device = "cpu"

Does anybody have a clue what might cause this issue and how I can circum-/prevent it?

Any error logs?

No, thats the thing, I am not getting any exceptions thrown or anything

Is the container pod restarting? How do you tell it runs init again?

Copied from the logs the following lines are printed behind one another:

> Starting Prediction

> Starting Setup

> Concluding Setup

While the container seems to stay running, as the initial info logging below is only shown once in the very beginning at startup: [INFO] [cli] Starting production HTTP BentoServer from "service:Model" listening on http://localhost:3000 (Press CTRL+C to quit)

Could you also share the bento service definition code?

Sure. I've been playing around with the resource configuration a bit, but this is the most recent version

@bentoml.service(
    resources={
        "cpu": "1",
        "memory": "2G",
        },
    traffic={"timeout": 600},
    http={"port": 8000},
)
class Model:
    def __init__(self) -> None:
        self.model = init_model()
       
    @bentoml.api()
    def execute(
            self,
            image: PILImage = bentoml.Field(description="Image to be processed")
    ):
        return model_exec(self.model, image)

where is the “Starting Prediction” printed?

Ah sorry, I just now saw your message. Its printed at the very beginning of _model_exec_

Do you observe the same behavior when running the container locally?

No, locally it does not show this behavior

Did you happen to set the “workers” count?

no, not explicitly

Could you share the relevant event logs from the scheduler?

Can you let me know how to retrieve them?

maybe ask your kubernetes admin for help? this seems like not an issue with BentoML framework

Using kubectl I am not able to see any relevant events, as k8s are all okay and no events occur when the request is being processed. The latest event remains: 51s Normal Started Pod/invoice-docvqa-demo-c575cccc4-7vwls Started container invoice-docvqa-demo

Could you share the full container logs?

Yeah sure, here are the logs logged in the container:

2024-06-18T06:02:29+0000 [INFO] [cli] Starting production HTTP BentoServer from "service:Model" listening on <http://localhost:3000> (Press CTRL+C to quit)
>> Starting Setup
>> Concluding Setup
>> Starting Prediction
>> Starting Setup
>> Concluding Setup
2024-06-18T06:05:09+0000 [INFO] [entry_service:Model:1] _ (scheme=http,method=CONNECT,path=109.71.252.64:1337,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 5.746ms (trace=4fc0f4c38db2060eb47bfd707827b48f,span=3206645bb1e6d3cb,sampled=0,service.name=Model)
2024-06-18T06:05:09+0000 [INFO] [entry_service:Model:1] _ (scheme=http,method=CONNECT,path=109.71.252.64:1337,type=,length=) (status=404,type=text/plain; charset=utf-8,length=9) 3.415ms (trace=5c4acce1e8ac57b3a50f6da030ddf757,span=91aaead9155c0572,sampled=0,service.name=Model)

And these are the events in k8s:

2m33s    Normal  Scheduled  Pod/invoice-docvqa-demo-59d6cd68cc-jjvtg  Successfully assigned namespace/invoice-docvqa-demo-59d6cd68cc-jjvtg to aks-workload-31352444-vmss000000
2m33s    Normal  Pulling   Pod/invoice-docvqa-demo-59d6cd68cc-jjvtg  Pulling image "invoice-docvqa-demo:0.0.10"
85s     Normal  Pulled   Pod/invoice-docvqa-demo-59d6cd68cc-jjvtg  Successfully pulled image "invoice-docvqa-demo:0.0.10" in 1m8.7737941s (1m8.773868699s including waiting)
85s     Normal  Created   Pod/invoice-docvqa-demo-59d6cd68cc-jjvtg  Created container invoice-docvqa-demo
84s     Normal  Started   Pod/invoice-docvqa-demo-59d6cd68cc-jjvtg  Started container invoice-docvqa-demo

And here again the current code of service.py:

@bentoml.service(
   resources={
       "cpu": "1",
       "memory": "2G",
       },
   traffic={"timeout": 600},
   http={"port": 8000},
)
class Model:
   def __init__(self) -> None:
       self.model = init_model()
     
   @bentoml.api()
   def execute(
           self,
           image: PILImage = bentoml.Field(description="Image to be processed")
   ):
       return model_exec(self.model, image)

And the actual implementation in model.py:

class Model_dto(BaseModel):
   model: VisionEncoderDecoderModel
   device: str
   processor: DonutProcessor
  
   class Config:
       arbitrary_types_allowed = True

def init_model():
   print(">> Starting Setup")
   processor = DonutProcessor.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
   model = VisionEncoderDecoderModel.from_pretrained("naver-clova-ix/donut-base-finetuned-docvqa")
   device = "cuda" if torch.cuda.is_available() else "cpu"
   <http://model.to|model.to>(device)
  
   model_dto = Model_dto(model=model, device=device, processor=processor)
   print(">> Concluding Setup")
   return model_dto

def model_exec(model_dto: Model_dto, image: PILImage):
   print(">> Starting Prediction")
   model = model_dto.model
   device = model_dto.device
   processor = model_dto.processor
  
   task_prompt = "<s_docvqa><s_question>{user_input}</s_question><s_answer>"
   prompt = task_prompt.replace("{user_input}", "What is the invoice total?")
   decoder_input_ids = processor.tokenizer(prompt, add_special_tokens=False, return_tensors="pt").input_ids

   pixel_values = processor(image, return_tensors="pt").pixel_values

   try:
       outputs = model.generate(
           <http://pixel_values.to|pixel_values.to>(device),
           decoder_input_ids=<http://decoder_input_ids.to|decoder_input_ids.to>(device),
           max_length=model.decoder.config.max_position_embeddings,
           pad_token_id=processor.tokenizer.pad_token_id,
           eos_token_id=processor.tokenizer.eos_token_id,
           use_cache=True,
           bad_words_ids=[[processor.tokenizer.unk_token_id]],
           return_dict_in_generate=True,
       )

       sequence = processor.batch_decode(outputs.sequences)[0]
       sequence = sequence.replace(processor.tokenizer.eos_token, "").replace(processor.tokenizer.pad_token, "")
       sequence = re.sub(r"<.*?>", "", sequence, count=1).strip()  # remove first task start token
       print(">> Concluding Prediction")
       return processor.token2json(sequence)
   except Exception as e:
       print("model generate error", e)
       return None

Sorry I didn’t follow up sooner, based on the logs, it seems insufficient resources - when your python process exceeds the memory limit set on your kubernetes pod specs, Kubernetes will try kill and restart the container