This message was deleted BentoML #ask-for-help

Join Slack

This message was deleted.

# ask-for-help

Slackbot

09/13/2023, 10:13 PM

This message was deleted.

Brian Marks

09/14/2023, 6:23 PM

@Jian Shen Yap apologies for the ping, but while you're around, have you seen this sort of behavior on

yatai

before? I'm happy to provide more detail as well if this question wasn't clear enough.

Jian Shen Yap

09/14/2023, 6:40 PM

Hey! No worries, any reason that you are using

v1alpha3

but not

v2alpha1

API?

Jian Shen Yap

09/14/2023, 6:41 PM

also you may need to do "axiom-model:tygqrysso6aiwlsr" as

is a keyword in yaml expression

Brian Marks

09/14/2023, 6:48 PM

Hey! No worries, any reason that you are using
v1alpha3
but not
v2alpha1
API?

That was from the example I was using. I'll upgrade! 🙂

Brian Marks

09/14/2023, 7:01 PM

Yeah, unfortunately that doesn't seem to fix the issue. I changed my

yatai.yml

to match the example in the BentoML README:

Copy code

apiVersion: <http://resources.yatai.ai/v1alpha1|resources.yatai.ai/v1alpha1>
kind: BentoRequest
metadata:
    name: axiom-model
    namespace: yatai
spec:
    bentoTag: "axiom-model:tygqrysso6aiwlsr"
---
apiVersion: <http://serving.yatai.ai/v2alpha1|serving.yatai.ai/v2alpha1>
kind: BentoDeployment
metadata:
    name: my-bento-deployment
    namespace: yatai
spec:
    bento: axiom-model
    ingress:
        enabled: true
    resources:
        ...
    runners:
        - name: axiom-model
          resources:
              ...
          autoscaling:
              ...

Ran it successfully in terminal:

Copy code

❯ kubectl apply -f yatai.yml    
<http://bentorequest.resources.yatai.ai/axiom-model|bentorequest.resources.yatai.ai/axiom-model> configured
<http://bentodeployment.serving.yatai.ai/my-bento-deployment|bentodeployment.serving.yatai.ai/my-bento-deployment> unchanged

But nothing changes in the UI. No deployment was started as a result of running the

kubectl apply

Jian Shen Yap

09/14/2023, 7:23 PM

could you try tearing down everything and do a fresh apply?

👍 1

Brian Marks

09/14/2023, 10:11 PM

Yeah, following a full redeploy it's the same situation:

Copy code

❯ kubectl apply -f yatai.yml
<http://bentorequest.resources.yatai.ai/axiom-model|bentorequest.resources.yatai.ai/axiom-model> created
<http://bentodeployment.serving.yatai.ai/my-bento-deployment|bentodeployment.serving.yatai.ai/my-bento-deployment> created

Copy code

❯ cat yatai.yml
apiVersion: <http://resources.yatai.ai/v1alpha1|resources.yatai.ai/v1alpha1>
kind: BentoRequest
metadata:
    name: axiom-model
    namespace: yatai
spec:
    bentoTag: "axiom-model:tygqrysso6aiwlsr"
---
apiVersion: <http://serving.yatai.ai/v2alpha1|serving.yatai.ai/v2alpha1>
kind: BentoDeployment
metadata:
    name: my-bento-deployment
    namespace: yatai
spec:
    bento: axiom-model
    ingress:
        enabled: true
    resources:
        limits:
            cpu: "500m"
            memory: "512m"
        requests:
            cpu: "250m"
            memory: "128m"
    autoscaling:
        maxReplicas: 10
        minReplicas: 2
    runners:
        - name: axiom-model
          resources:
              limits:
                  cpu: "1000m"
                  memory: "1Gi"
              requests:
                  cpu: "500m"
                  memory: "512m"
          autoscaling:
              maxReplicas: 4
              minReplicas: 1

No deployments.

Brian Marks

09/14/2023, 10:15 PM

Worth noting that its able to retrieve the bento successfully when the bento tag has no integers in it!

Copy code

[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [CheckingImage] Checking image exists: 127.0.0.1:5000/yatai-bentos:yatai.axiom-model.abdsicstjscaqlsr
[2023-09-14 15:14:38] [BentoDeployment] [klknjk] [Reconciling] Starting to reconcile BentoDeployment
[2023-09-14 15:14:38] [BentoDeployment] [klknjk] [GetBento] Getting Bento axiom-model--abdsicstjscaqlsr
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [CheckingImage] Image not exists: 127.0.0.1:5000/yatai-bentos:yatai.axiom-model.abdsicstjscaqlsr
[2023-09-14 15:14:38] [BentoDeployment] [klknjk] [GetBento] Bento axiom-model--abdsicstjscaqlsr not found
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Making sure docker config secret yatai-regcred in namespace yatai
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Getting bento axiom-model:abdsicstjscaqlsr from yatai service
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Got bento axiom-model:abdsicstjscaqlsr from yatai service
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Getting secret yatai-api-token in namespace yatai
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Secret yatai-api-token is found in namespace yatai, so updating it
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Secret yatai-api-token is updated in namespace yatai
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Getting model axiom-model:5hnxwqsso2o32lsr from yatai service
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Model axiom-model:5hnxwqsso2o32lsr is got from yatai service
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderJob] Creating image builder job: yatai-bento-image-builder-ck1objgo112s73btq870
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderJob] Created image builder job: yatai-bento-image-builder-ck1objgo112s73btq870
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] (combined from similar events): Making sure docker config secret yatai-regcred in namespace yatai
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [GenerateImageBuilderPod] Docker config secret yatai-regcred in namespace yatai is ready
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [CheckingImageBuilderJob] Found image builder job: yatai-bento-image-builder-ck1objgo112s73btq870
[2023-09-14 15:14:38] [BentoRequest] [axiom-model--abdsicstjscaqlsr] [CheckingImageBuilderJob] Found 0 pods for image builder job: yatai/yatai-bento-image-builder-ck1objgo112s73btq870

Brian Marks

09/14/2023, 10:18 PM

It still fails to build the image for some other reason, though

Jian Shen Yap

09/15/2023, 8:36 AM

@Jiang do you mind taking a look at this?

Jiang

09/15/2023, 8:43 AM

@Jian Shen Yap Maybe send to the cloud channel for help? I'm working on another support thread

👍 1

Brian Marks

09/18/2023, 4:09 PM

Which cloud channel are you referring to? I may not have access.

Jian Shen Yap

09/20/2023, 6:21 AM

@Xipeng Guan have you seen this before?

Xipeng Guan

09/20/2023, 6:33 AM

You can try to update yatai-image-builder to the latest version.

3 Views

Open in Slack

Previous Next