This message was deleted.
# ask-for-helps
Slackbot
10/20/2022, 7:44 AMThis message was deleted.
s
Sean
10/20/2022, 5:09 PMHi @Benjamin Tan, what errors did you see with GCR?
b
Benjamin Tan
10/20/2022, 5:11 PMSo I'm setting it up using workload identity
Benjamin Tan
10/20/2022, 5:12 PMBut getting
Copy code
1.666285667083234e+09 DEBUG events Failed to reconcile BentoDeployment: failed to check image <http://asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q|asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q> exists for bento ktp_ocr:i6xjufco5o6p237q: create docker registry client for <https://asia.gcr.io>: Get "<https://asia.gcr.io/v2/>": http: non-successful response (status=401 body="{\"errors\":[{\"code\":\"UNAUTHORIZED\",\"message\":\"Unauthorized access.\"}]}") {"type": "Warning", "object": {"kind":"BentoDeployment","namespace":"yatai","name":"ktp-ocr","uid":"a5cca888-0c13-4159-a896-e23b1519f2ab","apiVersion":"<http://serving.yatai.ai/v1alpha3|serving.yatai.ai/v1alpha3>","resourceVersion":"1520285427"}, "reason": "ReconcileError"}
1.6662856670832e+09 ERROR Reconciler error {"controller": "bentodeployment", "controllerGroup": "<http://serving.yatai.ai|serving.yatai.ai>", "controllerKind": "BentoDeployment", "BentoDeployment": {"name":"ktp-ocr","namespace":"yatai"}, "namespace": "yatai", "name": "ktp-ocr", "reconcileID": "a45f7585-c380-4ae5-8fcd-446ce417c01f", "error": "failed to check image <http://asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q|asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q> exists for bento ktp_ocr:i6xjufco5o6p237q: create docker registry client for <https://asia.gcr.io>: Get \"<https://asia.gcr.io/v2/>\": http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"Unauthorized access.\\\"}]}\")", "errorVerbose": "failed to check image <http://asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q|asia.gcr.io/jago-bank-data-production/yatai/yatai-bentos:yatai.ktp_ocr.i6xjufco5o6p237q> exists for bento ktp_ocr:i6xjufco5o6p237q: create docker registry client for <https://asia.gcr.io>: Get \"<https://asia.gcr.io/v2/>\": http: non-successful response (status=401 body=\"{\\\"errors\\\":[{\\\"code\\\":\\\"UNAUTHORIZED\\\",\\\"message\\\":\\\"Unauthorized access.\\\"}]}\")\<http://ngithub.com/bentoml/yatai-common/sync/errsgroup.(*Group).Wait\n\t/go/pkg/mod/github.com/bentoml/yatai-common@v0.0.0-20220913011805-6c642712fde7/sync/errsgroup/errsgroup.go:70\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).generatePodTemplateSpec\n\t/workspace/controllers/bentodeployment_controller.go:1457\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).generateDeployment\n\t/workspace/controllers/bentodeployment_controller.go:960\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).createOrUpdateDeployment\n\t/workspace/controllers/bentodeployment_controller.go:504\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).Reconcile\n\t/workspace/controllers/bentodeployment_controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594%22|ngithub.com/bentoml/yatai-common/sync/errsgroup.(*Group).Wait\n\t/go/pkg/mod/github.com/bentoml/yatai-common@v0.0.0-20220913011805-6c642712fde7/sync/errsgroup/errsgroup.go:70\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).generatePodTemplateSpec\n\t/workspace/controllers/bentodeployment_controller.go:1457\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).generateDeployment\n\t/workspace/controllers/bentodeployment_controller.go:960\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).createOrUpdateDeployment\n\t/workspace/controllers/bentodeployment_controller.go:504\ngithub.com/bentoml/yatai-deployment/controllers.(*BentoDeploymentReconciler).Reconcile\n\t/workspace/controllers/bentodeployment_controller.go:209\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:121\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:320\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:273\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.13.0/pkg/internal/controller/controller.go:234\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1594">}Benjamin Tan
10/20/2022, 5:13 PMSo basically unathorized access
Benjamin Tan
10/20/2022, 5:13 PMI've set workload identity on
yatai-deploymentyatai-builders yatai-system and yataiBenjamin Tan
10/20/2022, 5:17 PM Copy code
export DOCKER_REGISTRY_SERVER=<http://asia.gcr.io/jago-bank-data-production/yatai|asia.gcr.io/jago-bank-data-production/yatai>
export DOCKER_REGISTRY_USERNAME=''
export DOCKER_REGISTRY_PASSWORD=''
export DOCKER_REGISTRY_SECURE=true
export DOCKER_REGISTRY_BENTO_REPOSITORY_NAME=yatai-bentosBenjamin Tan
10/20/2022, 5:19 PMnot sure what else i should try
Benjamin Tan
10/21/2022, 3:00 AMSoooo something interesting
Benjamin Tan
10/21/2022, 3:01 AMI verified that workload identity works in all the `yatai`/`yatai-deployment` etc namespaces / service accounts
Benjamin Tan
10/21/2022, 3:03 AMThen I spun up a pod in one of these namespaces and shelled into it:
Copy code
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" "<https://asia.gcr.io/v2/>"
{"errors":[{"code":"UNAUTHORIZED","message":"Unauthorized access."}]}Benjamin Tan
10/21/2022, 3:04 AM Copy code
curl -H "Authorization: Bearer $(gcloud auth print-access-token)" "<https://asia.gcr.io/v2/_catalog>" Copy code
{"repositories":["jago-data-sandbox/6803a139-5ee9-44e0-a1aa-fe485824e4a9" ...3 Views