@kapa.ai I have this issue when creating a new connection:

Internal message: io.airbyte.workload.launcher.pipeline.stages.model.StageError: io.airbyte.workers.exception.KubeClientException: Failed to create pod source-s3-discover-ef1634a3-62ac-4bd0-8973-b8a309261753-0-gqnoe. at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:49) at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:40) at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.$$access$$apply(Unknown Source) at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Exec.dispatch(Unknown Source) at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456) at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:134) at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.doIntercept(InstrumentInterceptorBase.kt:65) at io.airbyte.metrics.interceptors.InstrumentInterceptorBase.intercept(InstrumentInterceptorBase.kt:48) at io.micronaut.aop.chain.MethodInterceptorChain.proceed(MethodInterceptorChain.java:143) at io.airbyte.workload.launcher.pipeline.stages.$LaunchPodStage$Definition$Intercepted.apply(Unknown Source) at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.apply(LaunchPodStage.kt:28) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:132) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onNext(MonoFlatMap.java:158) at reactor.core.publisher.Operators$ScalarSubscription.request(Operators.java:2571) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.MonoFlatMap$FlatMapMain.request(MonoFlatMap.java:194) at reactor.core.publisher.Operators$MultiSubscriptionSubscriber.set(Operators.java:2367) at reactor.core.publisher.FluxOnErrorResume$ResumeSubscriber.onSubscribe(FluxOnErrorResume.java:74) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.MonoFlatMap$FlatMapMain.onSubscribe(MonoFlatMap.java:117) at reactor.core.publisher.FluxFlatMap.trySubscribeScalarMap(FluxFlatMap.java:193) at reactor.core.publisher.MonoFlatMap.subscribeOrReturn(MonoFlatMap.java:53) at reactor.core.publisher.Mono.subscribe(Mono.java:4560) at reactor.core.publisher.FluxFlatMap$FlatMapMain.onNext(FluxFlatMap.java:430) at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:446) at reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:533) at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84) at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) at java.base/java.lang.Thread.run(Thread.java:1583) Caused by: io.airbyte.workers.exception.KubeClientException: Failed to create pod source-s3-discover-ef1634a3-62ac-4bd0-8973-b8a309261753-0-gqnoe. at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar$io_airbyte_airbyte_workload_launcher(KubePodClient.kt:255) at io.airbyte.workload.launcher.pods.KubePodClient.launchDiscover(KubePodClient.kt:211) at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:51) at io.airbyte.workload.launcher.pipeline.stages.LaunchPodStage.applyStage(LaunchPodStage.kt:28) at io.airbyte.workload.launcher.pipeline.stages.model.Stage.apply(Stage.kt:45) ... 44 more Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: <https://172.20.0.1:443/api/v1/namespaces/airbyte-dev/pods/source-s3-discover-ef1634a3-62ac-4bd0-8973-b8a309261753-0-gqnoe?fieldManager=fabric8>. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.KubernetesClientException.copyAsCause(KubernetesClientException.java:205) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:507) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handleResponse(OperationSupport.java:524) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:419) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.handlePatch(OperationSupport.java:397) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.handlePatch(BaseOperation.java:764) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.lambda$patch$2(HasMetadataOperation.java:231) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:236) at io.fabric8.kubernetes.client.dsl.internal.HasMetadataOperation.patch(HasMetadataOperation.java:251) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:1179) at io.fabric8.kubernetes.client.dsl.internal.BaseOperation.serverSideApply(BaseOperation.java:98) at io.airbyte.workload.launcher.pods.KubePodLauncher.create$lambda$0(KubePodLauncher.kt:58) at io.airbyte.workload.launcher.pods.KubePodLauncher$runKubeCommand$1.get(KubePodLauncher.kt:325) at dev.failsafe.Functions.lambda$toCtxSupplier$11(Functions.java:243) at dev.failsafe.Functions.lambda$get$0(Functions.java:46) at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74) at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187) at dev.failsafe.FailsafeExecutor.call(FailsafeExecutor.java:376) at dev.failsafe.FailsafeExecutor.get(FailsafeExecutor.java:112) at io.airbyte.workload.launcher.pods.KubePodLauncher.runKubeCommand(KubePodLauncher.kt:323) at io.airbyte.workload.launcher.pods.KubePodLauncher.create(KubePodLauncher.kt:52) at io.airbyte.workload.launcher.pods.KubePodClient.launchConnectorWithSidecar$io_airbyte_airbyte_workload_launcher(KubePodClient.kt:252) ... 48 more Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: PATCH at: <https://172.20.0.1:443/api/v1/namespaces/airbyte-dev/pods/source-s3-discover-ef1634a3-62ac-4bd0-8973-b8a309261753-0-gqnoe?fieldManager=fabric8>. Message: Unauthorized. Received status: Status(apiVersion=v1, code=401, details=null, kind=Status, message=Unauthorized, metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Unauthorized, status=Failure, additionalProperties={}). at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:642) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.requestFailure(OperationSupport.java:622) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.assertResponseCode(OperationSupport.java:582) at io.fabric8.kubernetes.client.dsl.internal.OperationSupport.lambda$handleResponse$0(OperationSupport.java:549) at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:646) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.http.StandardHttpClient.lambda$completeOrCancel$10(StandardHttpClient.java:141) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.http.ByteArrayBodyHandler.onBodyDone(ByteArrayBodyHandler.java:51) at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:863) at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:841) at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:510) at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2179) at io.fabric8.kubernetes.client.okhttp.OkHttpClientImpl$OkHttpAsyncBody.doConsume(OkHttpClientImpl.java:134) ... 3 more
Failure origin: airbyte_platform

Where does it come from ? How should I resolve it ?

The error is coming from the workload launcher trying to PATCH the Kubernetes pod for your S3 discover job, and the Kubernetes API is returning 401 Unauthorized on that PATCH request: >

Failure executing: PATCH ... pods/source-s3-discover-... Message: Unauthorized. ... code=401, reason=Unauthorized

This is the same pattern described in recent Kubernetes-related issues where the workload launcher fails on a

patch

operation for check/discover pods, not on the initial create. [Failed pod check] There is also an open issue where this manifests in the UI as “An unexpected error occurred (HTTP 500)” and is tied to workload API authentication configuration in Kubernetes. [HTTP 500 issue] From the available information, the likely causes are: • The service account / RBAC used by the

airbyte-workload-launcher

(or equivalent workload component) does not have permission to patch pods in the target namespace; or • If you have workload API auth enabled, the token / header configuration is incorrect, so the launcher’s calls to the Kubernetes API are rejected with 401. The knowledge sources suggest the following remediation steps for Kubernetes + Helm deployments: 1. Check RBAC / service account permissions Ensure the service account used by the workload launcher has at least: ◦

get

,

list

,

watch

,

create

,

patch

,

delete

on

pods

in the Airbyte namespace. The 401 on PATCH strongly indicates missing or invalid auth for that operation. [Failed pod check] 2. If using workload API auth, verify env vars For clusters where workload API auth is enabled, users have reported that setting and validating these values can help: [HTTP 500 issue] ◦

API_AUTHORIZATION_ENABLED=true

◦

WORKLOAD_API_AUTH_HEADER

(e.g.

Authorization

) ◦

WORKLOAD_API_BEARER_TOKEN

(the token expected by the workload API) Make sure: ◦ These env vars are set in the workload launcher pod. ◦ The token is valid and accepted by your Kubernetes API / admission layer. ◦ You are aware of the known limitation that these tokens may not propagate to connector sidecar containers (mentioned in the same issue). 3. Confirm Helm / values.yaml configuration If you upgraded to Helm chart v2 or changed values, double‑check any settings under

global.workloads.containerOrchestrator

and

workloadLauncher

in your

values.yaml

, especially anything related to auth or service accounts. [Chart v2 upgrade] ------------------------- Because the logs you shared stop at the Kubernetes 401 and the knowledge sources don’t include your exact Helm values or RBAC manifests, I can’t point to a single misconfigured field. But concretely, to move forward: • Inspect the service account used by

airbyte-workload-launcher

and its ClusterRole/RoleBindings. • If you have any custom API gateway / auth in front of the Kubernetes API, verify that the launcher’s requests include the expected token/header. • If you’re using the workload API auth flags above, confirm they are set and correct as per your cluster’s expectations. If you can share (with secrets redacted) the service account + role/rolebinding used by the workload launcher and any relevant

values.yaml

snippets, that would allow a more precise mapping to the patterns described in the linked issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.