This message was deleted BentoML #ask-for-help

Join Slack

This message was deleted.

# ask-for-help

Slackbot

02/13/2023, 11:00 AM

This message was deleted.

Xipeng Guan

02/13/2023, 11:04 AM

First, you should follow the documentation to check if the yatai pod is working properly: https://docs.bentoml.org/projects/yatai/en/latest/installation/yatai.html#verify-the-yatai-installation If it works properly, then check the network of your k8s cluster:

Copy code

kubectl run test-network --rm --tty -i --restart='Never' \
    --namespace default \
    --image curlimages/curl \
    --command -- sh -c 'curl <http://yatai.yatai-system.svc.cluster.local/api/v1/current_org>'

The above command should return:

Copy code

{"message":"username in cookie is empty"}pod "test-network" deleted

gratitude thank you 1

Junsu Lee

02/13/2023, 11:09 AM

Thank you for the quick reply reply! I thought it was an issue caused by changing the namespace of yatai-system. I'll check the network and share it!

Junsu Lee

02/13/2023, 11:23 AM

When I checked the network, I got the following.

Copy code

 ~/yatai  kubectl run test-network --rm --tty -i --restart='Never' \
    --namespace default \
    --image curlimages/curl \
    --command -- sh -c 'curl <http://yatai.yatai-system.svc.cluster.local/api/v1/current_org>'

curl: (6) Could not resolve host: yatai.yatai-system.svc.cluster.local
pod "test-network" deleted
pod default/test-network terminated (Error)

Junsu Lee

02/13/2023, 11:24 AM

Copy code

Junsu Lee

02/13/2023, 11:26 AM

It seems that

yatai-system

in the URL means the

yatai-system-namespace

. If I make a request below, you get a normal response.

Copy code

'curl <http://yatai|http://yatai>.<yatai-system-namespace>.svc.cluster.local/api/v1/current_org'

Copy code

kubectl run test-network --rm --tty -i --restart='Never' \
    --namespace default \
    --image curlimages/curl \
    --command -- sh -c 'curl <http://yatai.yatai-system-beta.svc.cluster.local/api/v1/current_org|http://yatai.yatai-system-beta.svc.cluster.local/api/v1/current_org>'
{"message":"username in cookie is empty"}pod "test-network" deleted

Junsu Lee

02/13/2023, 11:52 AM

Is it impossible to register

yatai-deployment

if I change the namespace of

yatai-system

Xipeng Guan

02/13/2023, 12:15 PM

you should upgrade yatai-deployment to set the

yatai-system.namespace

to the correct value:

Copy code

helm -n yatai-deployment get values yatai-deployment > yatai-deployment-values.yaml

helm upgrade yatai-deployment bentoml/yatai-deployment -n yatai-deployment --values ./yatai-deployment-values.yaml --set yataiSystem.namespace=yatai-system-beta

kubectl -n yatai-deployment rollout restart deploy/yatai-deployment

Junsu Lee

02/13/2023, 12:23 PM

Even with that change, the URL is still called http://yatai.yatai-system.svc.cluster.local/api/v1/clusters/default/yatai_components

Xipeng Guan

02/13/2023, 12:23 PM

yatai-image-builder also need upgrade

Junsu Lee

02/13/2023, 12:26 PM

The image builder has already changed the yatai-system namespace as below.

Copy code

helm upgrade --install yatai-image-builder yatai-image-builder \
    --set yataiSystem.namespace=$SYSTEM_NAMESPACE \
    --repo <https://bentoml.github.io/helm-charts> \
    -n $IMAGE_BUILDER_NAMESPACE \
    --set dockerRegistry.server=$DOCKER_REGISTRY_SERVER \
    --set dockerRegistry.inClusterServer=$DOCKER_REGISTRY_IN_CLUSTER_SERVER \
    --set dockerRegistry.username=$DOCKER_REGISTRY_USERNAME \
    --set dockerRegistry.password=$DOCKER_REGISTRY_PASSWORD \
    --set dockerRegistry.secure=$DOCKER_REGISTRY_SECURE \
    --set dockerRegistry.bentoRepositoryName=$DOCKER_REGISTRY_BENTO_REPOSITORY_NAME

Xipeng Guan

02/13/2023, 12:27 PM

do you restart the yatai-image-builder deployment?

Junsu Lee

02/13/2023, 12:30 PM

yes

Xipeng Guan

02/13/2023, 12:31 PM

You should to check these values:

Copy code

helm get values yatai-image-builder -n yatai-image-builder

Junsu Lee

02/13/2023, 12:32 PM

Copy code

yataiSystem:
  namespace: yatai-system-beta

Xipeng Guan

02/13/2023, 12:32 PM

Copy code

kubectl -n yatai-system get secret yatai-common-env -o jsonpath='{.data.YATAI_ENDPOINT}' | base64 -d

Junsu Lee

02/13/2023, 12:33 PM

Copy code

 ~/yatai  helm get values yatai-deployment -n $DEPLOYMENT_NAMESPACE
USER-SUPPLIED VALUES:
USER-SUPPLIED VALUES: null
bentoDeploymentNamespaces:
- yatai-beta
layers:
  network:
    ingressClass: alb
yataiServiceAccount:
  namespace: yatai-system-beta
yataiSystem:
  namespace: yatai-system-beta

Junsu Lee

02/13/2023, 12:34 PM

Copy code

kubectl -n yatai-system get secret yatai-common-env -o jsonpath='{.data.YATAI_ENDPOINT}' | base64 -d

Copy code

 ~/yatai  kubectl -n $SYSTEM_NAMESPACE get secret yatai-common-env -o jsonpath='{.data.YATAI_ENDPOINT}' | base64 -d
<http://yatai.yatai-system.svc.cluster.local>

Xipeng Guan

02/13/2023, 4:17 PM

Copy code

kubectl -n yatai-system-beta get secret yatai-common-env -o jsonpath='{.data.YATAI_ENDPOINT}' | base64 -d

Xipeng Guan

02/13/2023, 4:50 PM

you should set this value to the correct value: https://github.com/bentoml/yatai-image-builder/blob/main/helm/yatai-image-builder/values.yaml#L75

Junsu Lee

02/13/2023, 9:56 PM

Copy code

helm upgrade yatai-image-builder bentoml/yatai-image-builder -n $IMAGE_BUILDER_NAMESPACE --values ./yatai-image-builder-values.yaml --set yataiSystem.namespace=$SYSTEM_NAMESPACE --set yatai.endpoint=<http://yatai>.$SYSTEM_NAMESPACE.svc.cluster.local

After applying, the yatai.endpoint has changed. However, it seems that there is a permission problem to get the api token. The error log says " `in the namespace \\\"yatai-system\\\"\"`".

Copy code

1.6763248225944798e+09	ERROR	registerYataiComponent	{"func": "registerYataiComponent", "error": "DoJsonRequest Error: [POST]<http://yatai.yatai-system-beta.svc.cluster.local/api/v1/clusters/default/yatai_components>: POST <http://yatai.yatai-system-beta.svc.cluster.local/api/v1/clusters/default/yatai_components> status=403, {\"message\":\"get api token: failed to get yatai config in cluster default in organization default: secrets \\\"yatai-common-env\\\" is forbidden: User \\\"system:serviceaccount:yatai-system-beta:yatai\\\" cannot get resource \\\"secrets\\\" in API group \\\"\\\" in the namespace \\\"yatai-system\\\"\"}"

Xipeng Guan

02/14/2023, 4:15 AM

you should restart yatai-image-builder and yatai-deployment after this upgrade

Junsu Lee

02/14/2023, 4:49 AM

I have already restarted with the command below. Still the same ERROR occurs

Copy code

kubectl -n $DEPLOYMENT_NAMESPACE rollout restart deploy/yatai-deployment
kubectl -n $IMAGE_BUILDER_NAMESPACE rollout restart deploy/yatai-image-builder

kubectl -n $DEPLOYMENT_NAMESPACE logs -f deploy/yatai-deployment

Xipeng Guan

02/14/2023, 8:49 AM

first, you should check the secret value:

Copy code

kubectl -n yatai-system-beta get secret yatai-common-env

Xipeng Guan

02/14/2023, 8:50 AM

second, you should set YATAI_SYSTEM_NAMESPACE=yatai-system-beta for

yatai

deployment in

yatai-system-beta

namespace

Junsu Lee

02/14/2023, 12:13 PM

Copy code

$ kubectl -n yatai-system-beta get secret yatai-common-env
NAME               TYPE     DATA   AGE
yatai-common-env   Opaque   2      36h

Junsu Lee

02/14/2023, 12:15 PM

How to set YATAI_SYSTEM_NAMESPACE=yatai-system-beta for yatai deployment

Junsu Lee

02/14/2023, 12:32 PM

What value does the secret value check for? There is no YATAI_SYSTEM_NAMESPACE value.

Copy code

kubectl -n yatai-system-beta get secret yatai-common-env

Junsu Lee

02/14/2023, 12:49 PM

Adding env YATAI_SYSTEM_NAMESPACE to deployment works fine! thank you

👀 1

Junsu Lee

02/14/2023, 12:49 PM

After restarting deployment, I also restarted image-builder, deployment.

42 Views

Open in Slack

Previous Next