Howdy! Running airbyte in a k8s cluster. I am wondering if there is a way to configure the k8s serviceaccount attached to the job pods invoked by the workers. Currently they use the “default” serviceAccount from the namespace defined in 

JOB_KUBE_NAMESPACE

 .

Hi. I'm trying to install Airbyte on a k3s setup I have. After installing using the helm chart from the git repo and defining the ingress in the values file I'm getting 404. I'm using an nginx ingress controller. Is there something else I need to do in terms of annotation or configuration to make Airbyte work using an ingress controller?

Hi all. I've installed airbyte via the helm chart, only supplying ingress host+tls+annotation values, and whenever i setup the first (mysql) source I get a

FileNotFoundException: /root/.kube/config

error in the second worker log. I don't have admin control over the GKE cluster that I'm using, so is there any setting expectations I should ask ops to check for? (ie, maybe stuff for mounting api server details??)

Is there a way to set up sources/destinations/connections from yaml files via k8s ?

Hi, Appreciate all the amazing work you guys are doing. We dived into Airbyte POC to bring data from MySQL into Snowflake with deployment in Kubernetes, but unfortunately our POC did not complete and we had to switch to Meltano with Singer Taps that worked successfully I am following the Github Issue created here https://github.com/airbytehq/airbyte/issues/9834 Had engaged in some conversations here https://airbytehq.slack.com/archives/C01MFR03D5W/p1643814115160219 https://airbytehq.slack.com/archives/C01MFR03D5W/p1643313227329700 https://airbytehq.slack.com/archives/C01MFR03D5W/p1643821033045139?thread_ts=1643814115.160219&cid=C01MFR03D5W The current problem is worse because the logs dont show anything, and the Source to Target hand-over shows no information. Only that the destination snowflake pod started and exited with -6 some code. No information about what parameters it was started with or which intermediate files it was working with. It does not indicate if my PV/PVC were full or having problems. Hopefully the github issue gets more eyes, and I’ll be sure to re-visit Airbyte again in future, but currently the Meltano pipeline is working. Thanks for creating Airbyte

For anyone struggling to get a K8 deployment up and running, check out plural.sh. It’s magical I had been using Airbyte deployed on an EC2 instance since August 2021 with 25+ active connections. The need for an SSH tunnel made it really tough for me to give access to non-technical teammates…which meant I became the bottleneck for managing syncs and adding new cnxns. And I shouldn't be - the Airbyte interface is super user-friendly. A plural deployment comes bundled with OIDC and OAuth, so your team can login to Airbyte with their Google account. it also gives you some flexibility if you're managing Airbyte programmatically from Dagster or similar - API service keys work out of the box check it out https://www.plural.sh

I see I can set the scheduler, server, and worker log level in the helm chart, can i do the same for sync jobs? I'm having issue with the source pod just deciding to close things down half way through and it's not recording why.

Hi guys do we know if we can create user accounts for Airbyte UI login access? Or is it like a hosted service which doesn't need oauth stuff?

I am new to airbyte so please ignore my naive question

Hey folks! It is possible to scale Airbyte in K8s with multiple temporal and worker pods?

Hi, having just joined this slack group - is it better to join discourse instead?

Hi there, I have Airbyte and Airflow deployed on K8s and now I want to use the AirbyteTriggerSyncOperator to trigger our syncs. However, when I run an airflow job to trigger a sync, I get the error:

404 Client Error: Not Found for url: https://<internal-url>/api/v1/api/v1/connections/sync

Where

https://<internal-url>

is the URL where the webapp is accessible from (exposed by a load balancer). Has anyone try a similar deployment and if so, how did you set up the airbyte connection in airflow? Thanks!

I Built a new source connector and built an docker images and its working fine. Now some other server running airbyte on kube, how to send/use my docker image on their server(airbyte on kube). Kindly help me.

Hi community! I created this issue with a PR on the Helm chart. The idea is to make PVCs optional when deploying to Kubernetes, adding

server.persistence.enabled

to

values.yaml

configuration options. Would love some feedback 🙏

Hi, we had to restart our kubernetes cluster running airbyte. Now airbyte temporal is failing because it’s still trying to connect with old cluster ip adresses. Does anyone know where this is cached? We tried deleting the whole temporal and temporal_vissibillity database, but it’s still looking at old IPs. We also tried clearing the cluster and restarting.

Hi team, i have hosted airbyte on kubernetes cluster, everything is up, and when i try to add a source as postgres , it just throws the errror , on finding the kubernetes pod , one of the pod stuck in init and eventually failed

Failed Pod : pod/urce-postgres-sync-a1bbe16a-00b1-41d7-be9e-6904f435a348-0-mtfks   0/4     Init:Error   0          2m53s

Worker pod logs

2022-05-06 11:58:24 INFO i.a.w.p.KubePodProcess(<init>):512 - Creating pod urce-postgres-sync-a1bbe16a-00b1-41d7-be9e-6904f435a348-0-mtfks...
Log4j2Appender says: Creating pod urce-postgres-sync-a1bbe16a-00b1-41d7-be9e-6904f435a348-0-mtfks...
2022-05-06 11:58:26 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):305 - Waiting for init container to be ready before copying files...
Log4j2Appender says: Waiting for init container to be ready before copying files...
2022-05-06 11:58:26 INFO i.a.w.p.KubePodProcess(waitForInitPodToRun):309 - Init container present..
Log4j2Appender says: Init container present..
2022-05-06 11:58:29 INFO i.a.w.t.TemporalAttemptExecution(lambda$getWorkerThread$2):161 - Completing future exceptionally...
io.airbyte.workers.WorkerException: Error while getting checking connection.
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:84) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:27) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:158) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:833) [?:?]
Caused by: io.airbyte.workers.WorkerException: An error has occurred.
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:148) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:58) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.36.9-alpha.jar:?]
	... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: An error has occurred.
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:103) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:97) ~[kubernetes-client-5.12.2.jar:?]
	at io.fabric8.kubernetes.client.dsl.internal.WatchConnectionManager.lambda$run$2(WatchConnectionManager.java:133) ~[kubernetes-client-5.12.2.jar:?]
	at java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:934) ~[?:?]
	at java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:911) ~[?:?]
	at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) ~[?:?]
	at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:2162) ~[?:?]
	at io.fabric8.kubernetes.client.okhttp.OkHttpWebSocketImpl$BuilderImpl$1.onFailure(OkHttpWebSocketImpl.java:72) ~[kubernetes-client-5.12.2.jar:?]
	at okhttp3.internal.ws.RealWebSocket.failWebSocket(RealWebSocket.java:571) ~[okhttp-3.12.12.jar:?]
	at okhttp3.internal.ws.RealWebSocket$2.onFailure(RealWebSocket.java:221) ~[okhttp-3.12.12.jar:?]
	at okhttp3.RealCall$AsyncCall.execute(RealCall.java:211) ~[okhttp-3.12.12.jar:?]
	at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32) ~[okhttp-3.12.12.jar:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) ~[?:?]
	... 1 more

can any one help on this ?

Hi, I had question about Running ELT in Kubernetes as pod If ELT is running in a POD, for a large dataset, The Source fetches data in batches, Does it need to persist those batch in FileSystem / PVC Or does it all happen in memory ? Can a job fill up the PVC or Memory if the load is too huge ? (edited)

hey everyone. I installed airbyte on k8s using guide. But I see some weird logs in

airbyte-worker

Caused by: io.airbyte.workers.WorkerException: Failure executing: POST at: <https://10.0.0.1/api/v1/namespaces/etl/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:etl:airbyte-admin" cannot create resource "pods" in API group "" in the namespace "etl": RBAC: <http://role.rbac.authorization.k8s.io|role.rbac.authorization.k8s.io> "airbyte-admin-role" not found.
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.38.3-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:80) ~[io.airbyte-airbyte-workers-0.38.3-alpha.jar:?]
	at io.airbyte.workers.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:53) ~[io.airbyte-airbyte-workers-0.38.3-alpha.jar:?]
	... 3 more
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.0.0.1/api/v1/namespaces/etl/pods>. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods is forbidden: User "system:serviceaccount:etl:airbyte-admin" cannot create resource "pods" in API group "" in the namespace "etl": RBAC: <http://role.rbac.authorization.k8s.io|role.rbac.authorization.k8s.io> "airbyte-admin-role" not found.

Any tips?

Hi guys, i'm running airbyte on K8s with the supplied helm chart but i was wondering about the service account that gets created. Is the airbyte-admin service account also used to write the logs to a storage bucket?

Looking for a reviewer for my PR to add annotations on the ClusterIP service

Hi I've deploy Airbyte onto my Kubernetes cluster. I'm testing a simple connection from Postgres to Local file. But I'm getting this error when I tried to sync.

java.lang.RuntimeException: io.airbyte.workers.WorkerException: Running the launcher replication-orchestrator failed

Hi guys! We have deployed airbyte in k8s. Our cluster policy is that we can't deploy pods that point to public images and also the pod should run as

non-root.

Have made the necessary changes to fulfill our cluster policy and all airbyte services have been successfully deployed and running. But now we have another blocker due to run as

non-root

policy. I'm adding a new connector from the UI (in order to point the image to our private registry) but the pod that gets deployed in order to add this new connector is failing to start due to

non-root

policy

Error: container has runAsNonRoot and image will run as root (pod: "source-mssql-spec-1dfbbc78-d22d-4cc6)

The image is trying to run as

root

The image is a copy of

airbyte/source-mssql:0.3.22

public image. Anyone has encounter this issue before and was able to resolve?

hi everyone. Probably simple question, but how would you allocate more memory/cpu for source/destination pods? I did

JOB_MAIN_CONTAINER_MEMORY_REQUEST

but I don't see it applied after I described newly launched pods. Was this the correct env?

I have a mix of x86 and ARM (Graviton2) nodes in my k8's cluster. It doesn’t look like

source-s3-check-*

is picking up my `worker.nodeSelector`and is getting scheduled on an ARM node and crashing. Is there somewhere else that I should be configuring the

nodeSelector

for these pods? (I’m using the helm chart to deploy airbyte)

anyone have success in running airbyte on kubernetes with container orchestration enabled? getting

java.lang.RuntimeException: io.temporal.serviceclient.CheckedExceptionWrapper: java.util.concurrent.ExecutionException: java.lang.RuntimeException: io.airbyte.workers.exception.WorkerException: Running the launcher replication-orchestrator failed

without container orchestration enabled syncs work but resetting a connection results in the destination not having resources specified, despite the fact that I specified resources in every place I could

Hi team! I tried running OSS Airbyte on a local cluster and everything was great, but we need to run it in our on-premises Kubernetes cluster, where we do NOT have cluster admin rights. And here, during the deployment process, Airbyte tries to create a Service Account and falls off with an error:

<http://roles.rbac.authorization.k8s.io|roles.rbac.authorization.k8s.io> "airbyte-admin-role" is forbidden: User "Devuser" cannot get resource "roles" in API group "<http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>"

Please tell me, is it possible to launch Airbyte without creating a separate Service Account? Can we skip this step by making changes to the manifest?

Hey all, we have been working to create a full GCP Terraform deployment of all components to form a full Airbyte stack: • Airbyte GKE Cluster • Airflow (Composer 2) • Cloud SQL (PostreSQL) • Cloud Storage • VPC & Firewall rules to support We had two issues come up we hope there might be some insight into. 1. When disabling Minio in the helm chart after defining a Cloud Storage destination, logs will not write to storage. a. After enabling Minio with the GCS setup logs are going to GCS 2. Helm Chart versioning is not being incremented with Airbyte versioning. a. This shows on line 18 the helm chart version, and on line 24 the app version. https://github.com/airbytehq/airbyte/blob/master/charts/airbyte/Chart.yaml#L15-L18 b. In our case the 'appVersion' was incremented without carrying forward a new helm chart version as well. Even on line 15 it says "This version number should be incremented each time you make changes to the chart and its templates, including the app version." - this was not done. i. This causes problems when you want to define in the Terraform build the deployment you're going after, as you'd need to push a new helm chart version to pick up the new appVersion. I've been accustomed to submitting github issues, should I do the same for these?

@abhi Thanks for the talk the other day. I have been using plural for months now and i’m a fan Question - Is the airbyte API publicly accessible when deployed via Plural? I see there is an additional subdomain setup within the

context.yaml

. If I setup my instance at

airbyte.my-org.onplural.sh

, and setup the api at

airbyte-api.my-org.onplural.sh

How do I interact with API? I tried to replicate the network calls that I see in Chrome dev tools (~~`api/v1/worspace-id/get-destinations` ) with postman. I keep getting redirects to the ‘sign in with plural’ page i assume I need to use the other api-specific hostname? and how do i handle authentication - service token from app.plural.com or something else?

Hi everyone! I have deployed airbyte using the official helm chart (using the default config). But when i try to create a new connection it throws the fallowing error:

The connection tests failed.

Internal Server Error: The Access Key Id you provided does not exist in our records. (Service: S3, Status Code: 403, Request ID: 16FBA5BD9BF3F037, Extended Request ID: null)

Does anyone have faced it before?