This message was deleted.

Did you configure yatai to use this ECR?

yes i configured. Then you mean i have to push to docker like this:

docker push <ECR url>/yatai.iris_classifier.pphwznbpi66zubxx

Is it right?

With yatai, user only needs to push bento to yatai

Understood, when I pushed with command, it is automatically pushed to ECR and yatai

bentoml push iris_classifier:latest

right?

It pushes the bento to yatai

Thanks, I checked, your tutorial example is working well. But when I deployed with my bento, it is not uploaded to ECR and cannot find like this:

[CheckingImage] Image not exists: <http://954040596412.dkr.ecr.ap-northeast-2.amazonaws.com/yatai-bentos:yatai.iris_model_svc.5uf2qxrqboleqbxx|954040596412.dkr.ecr.ap-northeast-2.amazonaws.com/yatai-bentos:yatai.iris_model_svc.5uf2qxrqboleqbxx>

So I tried to upload image to ECR with that tag name manually. But Yatai cannot find the image. My bento image is a little big(6.3G). Is it size issue?

the size should be ok, have you seen any errors in the image builder pod?

This is a normal event message. When yatai-image-builder detects that the image does not exist, it will create a pod to build this image.

Defaulted container "builder" out of: builder, bento-downloader (init), model-downloader-0 (init)
error checking push permissions -- make sure you entered the correct tag name, and that you are authenticated correctly, and try again: checking push permission for "<http://954040596412.dkr.ecr.ap-northeast-2.amazonaws.com/yatai-bentos:yatai.iris_model_svc.5uf2qxrqboleqbxx|954040596412.dkr.ecr.ap-northeast-2.amazonaws.com/yatai-bentos:yatai.iris_model_svc.5uf2qxrqboleqbxx>": POST <https://954040596412.dkr.ecr.ap-northeast-2.amazonaws.com/v2/yatai-bentos/blobs/uploads/>: DENIED: Your authorization token has expired. Reauthenticate and try again.

It show this error. It seems token issue. is it right?

ah yes, looks like it’s an issue with pushing to ECR

If you are using ECR, you need to authenticate with IAM Role because the password for ECR will automatically expire. The detailed instructions on how to use IAM Role can be found in the documentation: https://docs.yatai.io/en/latest/installation/yatai_image_builder.html#prepare-container-registry

Then , should i delete iam role and create again?

You can create a new IAM role according to the documentation.

Were you already using IAM role before? I was under the impression that you were using Auth tokens?

Yes I already have used IAM role. but i think it was expired when i deployed. I recreated new IAM role. so there is no credential issue. however yatai-bento-image-builder is

OOMKilled

status. I think it is memory issue. Is it right?

Could be, did you specify the resource limited for the image builder pod?

I didn't, is there a resource limit for image builder pod?

You can change the image builder pod resources limit in JSON editor

Can I know where that json file is?

In the yatai deployment form

when creating the deployment from the UI, you can edit advanced configs from the JSON editor UI

I configured like this:

{
  "cluster_name": "default",
  "name": "test2",
  "description": "",
  "targets": [
    {
      "type": "stable",
      "bento_repository": "iris_model_svc",
      "bento": "qqqddjbqek6okbxx",
      "config": {
        "bento_request_overrides": {
          "imageBuilderContainerResources": {
            "limits": {
              "memory": "10Gi"
            }
          }
        },
        "hpa_conf": {
          "min_replicas": 2,
          "max_replicas": 10
        },
        "resources": {
          "requests": {
            "cpu": "500m",
            "memory": "500Mi",
            "gpu": ""
          },
          "limits": {
            "cpu": "1000m",
            "memory": "1024Mi",
            "gpu": ""
          }
        },
        "envs": [],
        "runners": {
          "iris_model": {
            "resources": {
              "requests": {
                "cpu": "500m",
                "memory": "500Mi",
                "gpu": ""
              },
              "limits": {
                "cpu": "1000m",
                "memory": "1024Mi",
                "gpu": ""
              }
            },
            "hpa_conf": {
              "min_replicas": 2,
              "max_replicas": 10
            },
            "deployment_strategy": "RollingUpdate"
          }
        },
        "enable_ingress": true,
        "deployment_strategy": "RollingUpdate"
      }
    }
  ],
  "kube_namespace": "yatai"
}

However, yatai-bento-image-builder shows this log:

Defaulted container "builder" out of: builder, bento-downloader (init), model-downloader-0 (init)

This is normal, and you should use kubectl describe to see if the resources field of the pod is correct

Yes it was because of worker node memory. I used m5.large EC2 and it cannot support 10Gi memory. Thanks for your quick support!! ☺️