https://linen.dev logo
Join Slack
Powered by
Hi team! I tried running OSS Airbyte on a local cl...
# kubernetes
u
Hi team! I tried running OSS Airbyte on a local cluster and everything was great, but we need to run it in our on-premises Kubernetes cluster, where we do NOT have cluster admin rights. And here, during the deployment process, Airbyte tries to create a Service Account and falls off with an error:
Copy code
<http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "airbyte-admin-role" is forbidden: User "Devuser" cannot get resource "roles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>"
Please tell me, is it possible to launch Airbyte without creating a separate Service Account? Can we skip this step by making changes to the manifest?
Full log:
Copy code
ulan@wsl2:/mnt/c/Projects/airbyte$ kustomize build kube/overlays/stable | kubectl apply -f -
serviceaccount/airbyte-admin created
configmap/airbyte-env-82bhmh4g56 created
configmap/airbyte-temporal-dynamicconfig created
configmap/sweep-pod-script created
secret/airbyte-secrets-ttfdbcfh47 created
secret/gcs-log-creds created
service/airbyte-db-svc created
service/airbyte-minio-svc created
service/airbyte-temporal-svc created
deployment.apps/airbyte-db created
deployment.apps/airbyte-minio created
deployment.apps/airbyte-pod-sweeper created
deployment.apps/airbyte-server created
deployment.apps/airbyte-temporal created
deployment.apps/airbyte-webapp created
deployment.apps/airbyte-worker created
job.batch/airbyte-bootloader created
Error from server (Forbidden): error when retrieving current configuration of:
Resource: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Resource=roles", GroupVersionKind: "<http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>, Kind=Role"
Name: "airbyte-admin-role", Namespace: "dev-namespace"
from server for: "STDIN": <http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "airbyte-admin-role" is forbidden: User "Devuser" cannot get resource "roles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" in the namespace "dev-namespace"
Error from server (Forbidden): error when creating "STDIN": <http://rolebindings.rbac.authorization.k8s.io|rolebindings.rbac.authorization.k8s.io> is forbidden: User "Devuser" cannot create resource "rolebindings" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>" in the namespace "dev-namespace"
Error from server (Forbidden): error when creating "STDIN": services "airbyte-server-svc" is forbidden: exceeded quota: object-quota, requested: services.nodeports=1, used: services.nodeports=0, limited: services.nodeports=0
Error from server (Forbidden): error when creating "STDIN": services "airbyte-webapp-svc" is forbidden: exceeded quota: object-quota, requested: services.nodeports=1, used: services.nodeports=0, limited: services.nodeports=0
Error from server (Forbidden): error when creating "STDIN": persistentvolumeclaims "airbyte-minio-pv-claim" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 200Mi
Error from server (Forbidden): error when creating "STDIN": persistentvolumeclaims "airbyte-volume-configs" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi
Error from server (Forbidden): error when creating "STDIN": persistentvolumeclaims "airbyte-volume-db" is forbidden: minimum storage usage per PersistentVolumeClaim is 1Gi, but request is 500Mi
Attaching a file with my permissions 
$ kubectl auth can-i --list
kubectl_auth_can-i_list.txt
g
You need a service account so that the container orchestrator can spin up the source and destination pods.
🙏 1
My suggestion would be to ask your k8s admin to allow some process to provision a service account if they don't want to allow users to. For example we have a deployment process that can create service accounts that have more permissions than the users do
👀 1
u
Hi @Guy Feldman, Thank you very much for your answer! I've asked our k8s admin to add permissions in addition to the current permissions of our service account, in particular these:
Copy code
rules:
  - apiGroups: ["*"]
    resources: ["jobs", "pods", "pods/log", "pods/exec", "pods/attach"]
    verbs: ["get", "list", "watch", "create", "update", "patch", "delete"] # over-permission for now
and they said: the service account should already have that for jobs, pods and pods/logs. It does not have access to pods/exec or pods/attach and we will not be able to grant that. Regular users do have access to pods/exec and pods/attach for troubleshooting purposes.
Maybe we can run Airbyte with limited rights, and for example, disable some of its features that require extended rights?
g
I am pretty sure the exec and attach are used to copy configurations into the pods and would require a major rewrite
perhaps do a docker deployment on a single machine to start?
u
Do you mean to deploy a container with installed docker /docker-compose in a Kubernetes cluster and deploy Airbyte there?
Because I deployed everything locally - both through the docker compose and through the local Kubernetes, and everything worked fine.
But our connectors must work in an internal kubernetes cluster, the admins of which cannot give us additional rights...
g
I meant just a virtual machine with docker compose installed
u
Hi @Guy Feldman, thank you for your time! We had a meeting with customer engineers and management, they explained that the privileges that Airbyte needs ("pods/exec", "pods/attach") are undesirable for internal security reasons. They suggested trying to configure Airbyte differently, and if it doesn't work out, unfortunately, we will be forced to abandon Airbyte and look for another solution.
Hi @Guy Feldman! As a result, I tried to comment out the manifest with the creation of a role, and I managed to run Airbyte on our cluster. But I can't add a new connector, as you wrote earlier - due to the fact that you need to copy config-files. So, given that "kubectl cp" is just a wrapper around "kubectl exec", and also that you in Airbyte decided to "Use kubectl cp to copy files instead of kube java clients", I wanted to ask - if I rewrite the code and replace all "kubectl cp" back with "kube java clients", will this bypass the restrictions on our local cluster, no more actions require "pods/exec" privileges? Errors from my pods: 
INFO i.a.w.p.KubePodProcess(copyFilesToKubeConfigVolume):251 - Waiting for kubectl cp to complete
...
io.airbyte.workers.exception.WorkerException: Error while getting spec from image image_name
...
Caused by: io.airbyte.workers.exception.WorkerException: java.io.IOException: kubectl cp failed with exit code 1
from the logs of init container: 
Timeout while attempting to copy to init container, exiting with code 1...
44 Views
Open in Slack
PreviousNext
Powered by