BentoML documentation: https://docs.bentoml.org/en/latest/index.html Yatai documentation: https://docs.bentoml.org/projects/yatai/en/latest/index.html Yatai GitHub page: https://github.com/bentoml/Yatai Yatai application:

Sign-in URL: <http://default.cloud.bentoml.com/login>
Email: <mailto:ashley.zhao.dacml@gmail.com|ashley.zhao.dacml@gmail.com>
Password: iBQU2gnBtD

In here, @Xipeng Guan, @Jiang, @Sean are from Engineering. @Tim Liu is from Sales/Marketing/Product.

@Ashley Zhao is the technical writer we are working with to do a POC for Yatai documentation. 👋

Hello 😄 Nice to meet everyone

This message was deleted.

@Tim Liu Ashley will sync up with you on creating video contents for Yatai guides.

Hello again, It’s really nice to talk to everyone today 🙂 Please let me know if you have any questions/needs regarding the documentation, I’ll do my best to address them

This occurs after it’s preparing Kubernetes

Actually, this is resolved by downgrading K to 1.23.8

Done! kubectl is now configured to use "minikube2" cluster and "default" namespace by default

I do still run into the pod not healthy error:

kubectl describe pod yatai-76c48797d8-6pmng --namespace yatai-system
Name:             yatai-76c48797d8-6pmng
Namespace:        yatai-system
Priority:         0
Service Account:  yatai
Node:             minikube2/192.168.64.3
Start Time:       Thu, 22 Sep 2022 14:57:43 -0400
Labels:           <http://app.kubernetes.io/instance=yatai|app.kubernetes.io/instance=yatai>
                  <http://app.kubernetes.io/name=yatai|app.kubernetes.io/name=yatai>
                  pod-template-hash=76c48797d8
Annotations:      <none>
Status:           Pending
IP:               172.17.0.4
IPs:
  IP:           172.17.0.4
Controlled By:  ReplicaSet/yatai-76c48797d8
Containers:
  yatai:
    Container ID:  
    Image:         <http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>
    Image ID:      
    Port:          7777/TCP
    Host Port:     0/TCP
    Command:
      sh
      -c
    Args:
      exec /app/api-server serve -c /conf/config.yaml
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
    Restart Count:  0
    Liveness:       http-get http://:http/ delay=10s timeout=10s period=5s #success=1 #failure=15
    Readiness:      http-get http://:http/ delay=10s timeout=10s period=5s #success=1 #failure=60
    Environment Variables from:
      env         Secret  Optional: false
    Environment:  <none>
    Mounts:
      /conf from config (ro)
      /etc/localtime from host-timezone (ro)
      /var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-nbvxx (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  host-timezone:
    Type:          HostPath (bare host directory volume)
    Path:          /etc/localtime
    HostPathType:  
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  yatai
    Optional:    false
  kube-api-access-nbvxx:
    Type:                    Projected (a volume that contains injected data from multiple sources)
    TokenExpirationSeconds:  3607
    ConfigMapName:           kube-root-ca.crt
    ConfigMapOptional:       <nil>
    DownwardAPI:             true
QoS Class:                   BestEffort
Node-Selectors:              <none>
Tolerations:                 <http://node.kubernetes.io/not-ready:NoExecute|node.kubernetes.io/not-ready:NoExecute> op=Exists for 300s
                             <http://node.kubernetes.io/unreachable:NoExecute|node.kubernetes.io/unreachable:NoExecute> op=Exists for 300s
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  7m19s                  default-scheduler  Successfully assigned yatai-system/yatai-76c48797d8-6pmng to minikube2
  Warning  Failed     6m38s                  kubelet            Failed to pull image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>": rpc error: code = Unknown desc = Error response from daemon: Get "<https://quay.io/v2/>": dial tcp: lookup <http://quay.io|quay.io> on 192.168.64.1:53: read udp 192.168.64.3:43418->192.168.64.1:53: i/o timeout
  Warning  Failed     5m42s                  kubelet            Failed to pull image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>": rpc error: code = Unknown desc = Error response from daemon: Get "<https://quay.io/v2/>": dial tcp: lookup <http://quay.io|quay.io> on 192.168.64.1:53: read udp 192.168.64.3:48756->192.168.64.1:53: i/o timeout
  Warning  Failed     4m39s                  kubelet            Failed to pull image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>": rpc error: code = Unknown desc = Error response from daemon: Get "<https://quay.io/v2/>": dial tcp: lookup <http://quay.io|quay.io> on 192.168.64.1:53: read udp 192.168.64.3:58186->192.168.64.1:53: i/o timeout
  Normal   Pulling    3m57s (x4 over 7m18s)  kubelet            Pulling image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>"
  Warning  Failed     3m17s (x4 over 6m38s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m17s                  kubelet            Failed to pull image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>": rpc error: code = Unknown desc = Error response from daemon: Get "<https://quay.io/v2/>": dial tcp: lookup <http://quay.io|quay.io> on 192.168.64.1:53: read udp 192.168.64.3:58805->192.168.64.1:53: i/o timeout
  Warning  Failed     2m50s (x6 over 6m38s)  kubelet            Error: ImagePullBackOff
  Normal   BackOff    2m12s (x9 over 6m38s)  kubelet            Back-off pulling image "<http://quay.io/bentoml/yatai:1.0.0-a0|quay.io/bentoml/yatai:1.0.0-a0>"

If possible, perhaps I can meet up to walk through the installation steps together, should the errors appear to be something configuration related.

On second thought, the container doesn’t seem to be there. Let me try to fix the issue.

After some digging it looks like it isn’t pulling Kubelet images correctly, at least to me. Perhaps this has something to do with using Minikube on MacOS. Let me see if setting it up again helps. Otherwise since I never set up Kubernetes before, I may need to configure it in some other way?

I think it may have something to do with a VPN client on my mac. I’ll need to test this out in another OS other than my current

Hi Ashley, are you still blocked on any of these? We can find some time on Sunday evening to debug together.

Hi @Sean, Sure, do you have time now?

Yes. https://us02web.zoom.us/j/2069799684?pwd=Z2hJbEcxQmRYbEJNY1ZaL2dNQzJlZz09

I’m going to step out for a bit, actually.

Thanks, ok I will hop on whenever

Maybe coordinate and chat with @Xipeng Guan? Xipeng, what time is best for you?

Oh ok, I already scheduled a meeting with Xipeng tomorrow evening

This message was deleted.

Hi again, sorry ran into another issue it was while installing Yatai-deployment I was following instructions up till it hit helm chart configuration for yatai

kubectl -n yatai-deployment get pod -l app=docker-private-registry-proxy
NAME                                  READY   STATUS    RESTARTS   AGE
docker-private-registry-proxy-d9z2x   1/1     Running   0          6s
(base) ashleyzhao@Ashleys-MacBook-Pro ~ % kubectl -n yatai-deployment patch cm/network --type merge --patch '{"data":{"ingress-class":"$INGRESS_CLASS"}}'
Error from server (NotFound): configmaps "network" not found

Error also occurs in the public docker registry

Here is what I get for INGRESS_CLASS:

echo $INGRESS_CLASS

nginx

This message was deleted.

This message was deleted.

Rough overview of docs: • Get started ◦ Yatai components ▪︎ Yatai (Kubernetes, PostgreSQL protocol database, S3) ▪︎ Yatai deployment (cert-manager, metrics-server, docker registry, ingress class/annotations) ◦ Terminology/Additional resources (links) ◦ Minimum requirements ◦ Supported frameworks? (Everything is bento) ◦ Installation ▪︎ Yatai ▪︎ Yatai Deployment • Update • Tutorial • Yatai ◦ Manage models ◦ Deploy • Monitor ◦ Collecting metrics and logs • Uninstall? • Troubleshoot ◦ Limitations ▪︎ Does not support Docker on MacOS ◦ _Common installation questions (?)_First time set up AWS - give permissions ▪︎ Kubernetes problems - check VPN • Terminology/Additional information ◦ BentoML models ◦ Model registry ◦ Bento ◦ Bento registry ◦ Bento Deployment CRD I’ll share the Notion docs once I refine them. It’s in some very rough patches so I’m not sure if they’ll be ready to be reviewed. However you’re always welcome to take a look.

The only topic that I’m not sure if we need is an Uninstall section.

If there are other topics that need to be added/removed, feel free to let me know!

New format agree with Ashley.

## TOC

- Concepts
    - Architecture
        - Diagram ([<https://excalidraw.com/](https://excalidraw.com/)>)
    - Yatai (Kubernetes, PostgreSQL protocol database, S3)
    - Yatai deployment (cert-manager, metrics-server, docker registry, ingress class/annotations)
    - Observability (Prometheus, Grafana, Loki)
    - Terminology/Additional resources (links?)
        - BentoML models
        - Model registry
        - Bento
        - Bento registry
        - Bento Deployment CRD
- Installation
    - Yatai Installation/System requirements
    - [Yatai Deployment](<https://docs.bentoml.org/projects/yatai/en/latest/installation/yatai_deployment.html)/System> requirements
    - (Install) Observability (for Prometheus/Grafana/Loki)
    - [Update](<https://docs.bentoml.org/projects/yatai/en/latest/installation/yatai.html>) / Migration from Yatai 0.4
    - Troubleshoot
        - Limitations
            - Does not support Docker on MacOS
        - *Common installation questions (?)*
            - First time set up AWS - give permissions
            - Kubernetes problems - check VPN
    - Uninstall
- Tutorial (Video)
    - Yatai
        - Model and bento repository (push/pull and UI)
        - Administrator (manage users)
    - Yatai Deployment
        - Create, delete, update deployments from UI
        - Create, delete, update deployments from BentoDeployment CRD
            - Kubernetes
                - BentoDeployment CRD (Examples)
                    - Note: comes with `yatai-deployment`
                    
                    ```jsx
                    > kubectl apply deployment.yaml
                    > k get svc

- Observability - Metrics, Logs, (Traces) [Collecting metrics and logs](https://docs.bentoml.org/projects/yatai/en/latest/observability/metrics.html)```

Hi @Tim Liu, How are you? Just wondering if we have a video that covers the end-to-end user experience, specifically these points:

- Yatai (Dashboard overview)
    - Model and bento repository (push/pull and UI)
    - Administrator (manage users)

I can probably help since I’ve gone through a quick user experience. Can you connect with me/Sean to see if we can do a video together?