i am installing airbyte on GKE hitting issues here are my po Airbyte #replication-ask-ai

i am installing airbyte on GKE hitting issues here...

Shawna Zhao

12/30/2025, 6:09 AM

i am installing airbyte on GKE hitting issues here are my pods state airbyte-bootloader 0/1 Completed 0 2m2s airbyte-connector-builder-server-6cdb5bb655-5smmq 1/1 Running 0 79s airbyte-cron-6c4b76dcbd-88f6f 1/1 Running 0 79s airbyte-db-0 1/1 Running 0 85s airbyte-minio-0 0/1 Pending 0 3m3s airbyte-server-c6f5d5d4c-hvc2s 0/1 Running 0 77s airbyte-temporal-7fb46bd89d-4stn5 1/1 Running 0 79s airbyte-worker-5ccc858c87-6w69q 0/1 Running 0 77s airbyte-workload-api-server-b5f4bcf64-lr2dn 0/1 Running 0 79s airbyte-workload-launcher-667784c4b9-gdnff 0/1 Running 0 78s

kapa.ai

12/30/2025, 6:09 AM

From the pod list, the main red flag is

airbyte-minio-0

stuck in

Pending

. When MinIO can’t start or its volume can’t bind, Airbyte’s server and workers often show errors like connection refused to

airbyte-minio:9000

, and connectors can’t run. This matches a known pattern on Kubernetes/EKS where MinIO is pending due to volume binding issues. [[Minio pending](https://discuss.airbyte.io/t/1148)] The knowledge sources show that in similar cases,

kubectl describe pod airbyte-minio-...

reveals events like: > error while running "VolumeBinding" prebind plugin … Failed to bind volumes: timed out waiting for the condition [[Minio pending](https://discuss.airbyte.io/t/1148)] and the server logs show: > Unable to execute HTTP request: Connect to airbyte-minio:9000 … failed: Connection refused [[Minio logs](https://discuss.airbyte.io/t/1148)] ### What to check next 1. Describe the MinIO pod and PVC

Copy code

bash    kubectl describe pod airbyte-minio-0    kubectl get pvc -n <your-namespace>    kubectl describe pvc airbyte-minio -n <your-namespace>

You’re looking for volume binding / storage class / PV issues like in the example above. [[Minio pending](https://discuss.airbyte.io/t/1148)] 2. Check events in the namespace

Copy code

bash    kubectl get events -n <your-namespace> --sort-by='{.lastTimestamp}'

This often shows why the MinIO pod is pending (no suitable nodes, storage class misconfig, CSI driver issues, etc.). [[EKS timeout](https://discuss.airbyte.io/t/3127)] 3. Confirm storage configuration On other clusters (e.g., EKS) the root cause was that the dynamic provisioner (like

<http://ebs.csi.aws.com|ebs.csi.aws.com>

) wasn’t able to create/bind the volume in time. [[EKS timeout](https://discuss.airbyte.io/t/3127); [Minio pending](https://discuss.airbyte.io/t/1148)] On GKE, check that: - A default StorageClass exists and is usable. - Nodes have access to the disk type you’re using. - If you’re using a custom StorageClass in the Helm values, it matches what exists in the cluster. 4. Check server logs for MinIO-related errors

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-hvc2s

In similar cases, logs show S3/MinIO connection errors while MinIO is pending. [[Minio logs](https://discuss.airbyte.io/t/1148)] ### If you want to avoid MinIO One user switched logs to S3 and disabled MinIO via Helm values, but still saw MinIO pods because the chart condition was misconfigured in that version. [[Minio config](https://discuss.airbyte.io/t/1148)] The knowledge sources don’t provide a definitive, current Helm snippet for GKE to fully disable MinIO, so I can’t give a guaranteed-working config here. --- To move forward, please share: - Output of:

Copy code

bash   kubectl describe pod airbyte-minio-0   kubectl get pvc -n <your-namespace>   kubectl describe pvc airbyte-minio -n <your-namespace>   kubectl get events -n <your-namespace> --sort-by='{.lastTimestamp}'

Once we see the exact volume / scheduling error for

airbyte-minio-0

, we can pinpoint the fix.

Shawna Zhao

12/30/2025, 6:12 AM

@kapa.ai error i get from airbyte-server Warning Unhealthy 15s (x13 over 2m13s) kubelet Readiness probe failed: Get "http://10.80.1.27:8001/api/v1/health": dial tcp 10.80.1.278001 connect: connection refused Warning Unhealthy 15s (x6 over 105s) kubelet Liveness probe failed: Get "http://10.80.1.27:8001/api/v1/health": dial tcp 10.80.1.278001 connect: connection refused

kapa.ai

12/30/2025, 6:12 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:12 AM

Those messages mean the

airbyte-server

container is failing its HTTP health checks on

:8001/api/v1/health

, so Kubernetes keeps marking it Unhealthy and (for liveness) may restart it. This exact pattern appears when the server process is not actually listening yet or is stuck/crashing, so the probe’s TCP connection is refused. [[Quickstart issue](https://github.com/airbytehq/airbyte/issues/50430)] From the knowledge sources, there are two main categories to check: --- ### 1. Confirm the server is actually starting Get the server logs:

Copy code

bash kubectl logs airbyte-server-c6f5d5d4c-hvc2s

Look for: - Repeated startup failures / stack traces. - Errors connecting to dependencies (DB, Temporal, MinIO, etc.). For example, similar deployments showed

Connection refused

to Temporal or MinIO when those services weren’t healthy yet. [[K8s check conn](https://discuss.airbyte.io/t/594); [Minio logs](https://discuss.airbyte.io/t/1148)] If the logs show it never reaches “started” or is crashing in a loop, the readiness/liveness failures are a symptom, not the root cause. --- ### 2. Check dependencies that can block server startup From other Kubernetes installs, the server can be unhealthy because it can’t talk to required services: - MinIO: In your cluster

airbyte-minio-0

Pending

. In similar cases, the server logs show: > Unable to execute HTTP request: Connect to airbyte-minio:9000 … failed: Connection refused [[Minio logs](https://discuss.airbyte.io/t/1148)] And the MinIO pod is pending due to volume binding issues: > FailedScheduling … Failed to bind volumes: timed out waiting for the condition [[Minio pending](https://discuss.airbyte.io/t/1148)] Check MinIO and its PVC:

Copy code

bash   kubectl describe pod airbyte-minio-0   kubectl get pvc -n <namespace>   kubectl describe pvc airbyte-minio -n <namespace>   kubectl get events -n <namespace> --sort-by='{.lastTimestamp}'

If you see volume binding / storage class errors, you need to fix storage (StorageClass, CSI driver, permissions) so MinIO can start. Until MinIO is healthy, the server may fail to initialize fully. - Temporal / DB: Other GKE users saw “Cannot reach server” or failing checks when the server couldn’t connect to Temporal or the DB. [[K8s check conn](https://discuss.airbyte.io/t/594); [GKE no internet](https://discuss.airbyte.io/t/763)] Verify those pods are

Running

and check their logs if the server log shows connection errors. --- ### 3. Probe configuration The readiness/liveness probes for the server are standard HTTP checks on

/api/v1/health

. The chart allows tuning probe timings (initialDelaySeconds, timeout, etc.), but the knowledge source only shows an example snippet, not a specific fix for your case. [[Probe config](https://github.com/airbytehq/airbyte/issues/49843)] If logs show the server eventually becomes healthy but just needs more time, you could increase

initialDelaySeconds

and/or

timeoutSeconds

in your Helm values. However, if the server never binds to port 8001, changing probe timings won’t help; you must fix the underlying startup error. --- ### What to share next To narrow this down based on the documented patterns, please run and share:

Copy code

bash kubectl logs airbyte-server-c6f5d5d4c-hvc2s kubectl describe pod airbyte-minio-0 kubectl describe pvc -n <namespace>  # especially the MinIO PVC kubectl get events -n <namespace> --sort-by='{.lastTimestamp}'

The knowledge sources strongly suggest that a pending MinIO (or other dependency) plus connection-refused health checks on

:8001

usually means the server is failing to start because it can’t reach storage or Temporal, and fixing that dependency resolves the probe failures. [[Minio pending](https://discuss.airbyte.io/t/1148); [K8s check conn](https://discuss.airbyte.io/t/594)]

Shawna Zhao

12/30/2025, 6:13 AM

@kapa.ai i did kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test , there is no output

kapa.ai

12/30/2025, 6:13 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:14 AM

kubectl logs

for the server pod returns nothing, it usually means the container process is not getting far enough to emit logs (or is failing immediately), while Kubernetes is still trying to run it and probe it. The knowledge sources don’t describe this exact “no logs at all” case, but they do show very similar symptoms where: - The pod is

Running

but health probes fail with

connection refused

. [[Quickstart issue](https://github.com/airbytehq/airbyte/issues/50430)] - A dependency (MinIO, DB, Temporal, storage) is not healthy or its volume is not bound, causing the platform to be effectively unhealthy. [[Minio pending](https://discuss.airbyte.io/t/1148); [EKS timeout](https://discuss.airbyte.io/t/3127)] Given that: -

airbyte-minio-0

is still

Pending

in your cluster. -

airbyte-server

has failing readiness/liveness probes. -

kubectl logs

shows nothing from the server container. the most likely next step, based on the documented cases, is to focus on the MinIO/storage issue first. ### 1. Inspect MinIO and its PVC (this is where we do have guidance) From the MinIO-on-Kubernetes thread, a pending MinIO pod was caused by volume binding problems;

kubectl describe

showed events like: [[Minio pending](https://discuss.airbyte.io/t/1148)] > error while running "VolumeBinding" prebind plugin … Failed to bind volumes: timed out waiting for the condition Run:

Copy code

bash kubectl describe pod airbyte-minio-0 -n airbyte-test kubectl get pvc -n airbyte-test kubectl describe pvc airbyte-minio -n airbyte-test kubectl get events -n airbyte-test --sort-by='{.lastTimestamp}'

If you see messages about: -

FailedScheduling

with

VolumeBinding

errors, or -

waiting for a volume to be created, either by external provisioner ... or manually created by system administrator

then you’re hitting the same class of issue as in the MinIO thread and the EKS examples. [[Minio pending](https://discuss.airbyte.io/t/1148); [EKS timeout](https://discuss.airbyte.io/t/3127); [PVC pending](https://discuss.airbyte.io/t/1654)] You’ll need to fix the storage side (StorageClass, CSI driver, permissions) so the MinIO PVC can bind and the pod can start. ### 2. Check if the server container is actually starting Since

kubectl logs

shows nothing, also check the pod description to see container state:

Copy code

bash kubectl describe pod airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

Look for: -

State: Waiting

with a

Reason

CrashLoopBackOff

ImagePullBackOff

, or similar. - Any events about image pull failures or container startup failures. The knowledge sources don’t include a specific “no logs from server” troubleshooting path, so I can’t say more than: use

kubectl describe

to see why the container isn’t starting; that, plus the MinIO/storage errors, will point to the root cause. --- To proceed using what’s documented: 1. Share the output of: -

kubectl describe pod airbyte-minio-0 -n airbyte-test

kubectl describe pvc airbyte-minio -n airbyte-test

kubectl get events -n airbyte-test --sort-by='{.lastTimestamp}'

kubectl describe pod airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

2. Compare any MinIO PVC / volume errors with the examples above; fixing those has resolved similar “platform unhealthy” situations in other Kubernetes installs. [[Minio pending](https://discuss.airbyte.io/t/1148); [PVC pending](https://discuss.airbyte.io/t/1654)]

Shawna Zhao

12/30/2025, 6:16 AM

@kapa.ai

kapa.ai

12/30/2025, 6:16 AM

Sorry @Shawna Zhao, I'm not sure what you mean. Please try again. 👋

Shawna Zhao

12/30/2025, 6:18 AM

@kapa.ai i got log above for airbyte-server, and the pod failed after this

kapa.ai

12/30/2025, 6:18 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:18 AM

It looks like some of the context is missing (the “log above” from

airbyte-server

is not included in the messages I can see), so I can’t comment on the exact stack trace or error. I’ll outline what the knowledge sources do cover for “server pod fails after startup logs”, and how to use that to debug. From similar Kubernetes/GKE/EKS cases: 1. Server often fails because it can’t reach a dependency In other installs,

airbyte-server

fails to start (or crashes soon after) when it cannot connect to: - Temporal:

Copy code

text      finishConnect(..) failed: Connection refused: airbyte-temporal/10.x.x.x:7233

[[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655)] - MinIO / S3:

Copy code

text      Unable to execute HTTP request: Connect to airbyte-minio:9000 [...] failed: Connection refused

[[Minio logs](https://discuss.airbyte.io/t/1148)] - Or the database. When that happens, the server process exits, the pod fails, and Kubernetes reports readiness/liveness probe failures (connection refused on

:8001/api/v1/health

) just before the crash. [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] 2. You already have a known-bad dependency: MinIO is Pending In your cluster,

airbyte-minio-0

Pending

. In a documented case, MinIO was Pending because its PVC couldn’t bind: > error while running "VolumeBinding" prebind plugin … Failed to bind volumes: timed out waiting for the condition [[Minio pending](https://discuss.airbyte.io/t/1148)] That led to: - Server logs with S3/MinIO connection errors. - Connectors failing because logs/output storage was unavailable. [[Minio logs](https://discuss.airbyte.io/t/1148)] So even without seeing your exact server log, a very likely cause (and one we do have documentation for) is: server fails after startup because it can’t talk to MinIO, which is stuck Pending due to storage issues. 3. What to do next (based on documented patterns) a. Inspect MinIO and its PVC/events This is the main documented root cause for Pending MinIO and downstream failures:

Copy code

bash    kubectl describe pod airbyte-minio-0 -n airbyte-test    kubectl get pvc -n airbyte-test    kubectl describe pvc airbyte-minio -n airbyte-test    kubectl get events -n airbyte-test --sort-by='{.lastTimestamp}'

Compare what you see with the examples: -

FailedScheduling

with

VolumeBinding

errors. -

waiting for a volume to be created, either by external provisioner ...

[[Minio pending](https://discuss.airbyte.io/t/1148); [EKS timeout](https://discuss.airbyte.io/t/3127)] If that matches, you need to fix GKE storage (StorageClass, CSI driver, permissions) so the MinIO PVC can bind and the pod can start. b. Confirm what the server log is complaining about Since you say the pod fails after the log you saw, re-run:

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test --previous

(the

--previous

flag shows logs from the last crashed container). In other cases, this revealed explicit errors like “Connection refused: airbyte-temporal-svc:7233” or MinIO/S3 errors. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655); [Minio logs](https://discuss.airbyte.io/t/1148)] If your log matches one of those patterns, the fix is to make that dependency healthy (Temporal, DB, or MinIO). 4. Probe failures are a symptom, not the cause The readiness/liveness probe failures you saw:

Copy code

text    Readiness probe failed: Get "<http://10.80.1.27:8001/api/v1/health>": dial tcp ...: connect: connection refused    Liveness probe failed: Get "<http://10.80.1.27:8001/api/v1/health>": dial tcp ...: connect: connection refused

are exactly what we see when the server process is not listening on

:8001

because it crashed or never finished startup. [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] Adjusting probe timings alone won’t help if the underlying startup error (e.g., can’t reach MinIO/Temporal/DB) isn’t fixed. --- Because I don’t see your exact server log content, I can’t say with certainty which dependency is failing. But based on the knowledge sources and your pod states, the most likely path is: 1. Fix MinIO’s Pending state by resolving its PVC/StorageClass issue. [[Minio pending](https://discuss.airbyte.io/t/1148)] 2. Use

kubectl logs ... --previous

airbyte-server

to confirm whether it then starts cleanly or still fails due to another dependency (Temporal/DB). If you can paste the actual

airbyte-server

log lines you mentioned, I can map them more precisely to the documented issues and suggest a more targeted fix.

Shawna Zhao

12/30/2025, 6:21 AM

@kapa.ai the container doesnt seem to crash it is just killed Normal Pulled 2m3s kubelet Container image "airbyte/server:2.0.1" already present on machine Normal Killing 51s kubelet Container airbyte-server-container failed liveness probe, will be restarted Normal Pulled 46s (x2 over 107s) kubelet Container image "airbyte/server:2.0.1" already present on machine Normal Created 46s (x2 over 107s) kubelet

kapa.ai

12/30/2025, 6:21 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:21 AM

The events you’re seeing mean Kubernetes is repeatedly killing and restarting the

airbyte-server

container only because the liveness probe keeps failing:

Copy code

text Container airbyte-server-container failed liveness probe, will be restarted

This matches the pattern in other Airbyte installs where: - The probe hits

http://<pod-ip>:8001/api/v1/health

- The server is not responding (connection refused), so the probe fails - Kubelet kills and restarts the container in a loop [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] From the knowledge sources, the underlying reasons for this are almost always: 1. Server can’t reach a dependency and never becomes healthy - Temporal:

Connection refused: airbyte-temporal...:7233

[[Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] - MinIO/S3:

Connect to airbyte-minio:9000 [...] failed: Connection refused

[[Minio logs](https://discuss.airbyte.io/t/1148)] - DB: connection issues to

airbyte-db-svc

[[EKS timeout](https://discuss.airbyte.io/t/3127)] In your case,

airbyte-minio-0

is still

Pending

, which is a known cause of server-side S3/MinIO errors and platform instability. [[Minio pending](https://discuss.airbyte.io/t/1148)] 2. MinIO Pending due to storage / PVC problems In similar Kubernetes deployments, MinIO was Pending because its PVC couldn’t bind: > error while running "VolumeBinding" prebind plugin … Failed to bind volumes: timed out waiting for the condition [[Minio pending](https://discuss.airbyte.io/t/1148)] or: > waiting for a volume to be created, either by external provisioner … [[EKS timeout](https://discuss.airbyte.io/t/3127)] That led to: - Server logs complaining about S3/MinIO connection. - Jobs failing because logs/output storage was unavailable. [[Minio logs](https://discuss.airbyte.io/t/1148)] Given that, the liveness failures and restarts are a symptom. The root cause is almost certainly that the server never reaches a healthy state because a dependency (very likely MinIO) is not available. --- ### What to do next (based on documented cases) 1. Inspect MinIO and its PVC/events Run:

Copy code

bash kubectl describe pod airbyte-minio-0 -n airbyte-test kubectl get pvc -n airbyte-test kubectl describe pvc airbyte-minio -n airbyte-test kubectl get events -n airbyte-test --sort-by='{.lastTimestamp}'

Compare the output with the examples above. If you see

FailedScheduling

with

VolumeBinding

errors or “waiting for a volume to be created…”, you need to fix GKE storage (StorageClass, CSI driver, permissions) so the MinIO PVC can bind and the pod can start. [[Minio pending](https://discuss.airbyte.io/t/1148); [EKS timeout](https://discuss.airbyte.io/t/3127)] 2. Check the last crashed server logs Because the container is being killed by liveness, use `--previous`:

Copy code

bash kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test --previous

In other installs, this showed explicit errors like: -

Connection refused: airbyte-temporal-svc:7233

[[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655)] -

Unable to execute HTTP request: Connect to airbyte-minio:9000 [...] failed: Connection refused

[[Minio logs](https://discuss.airbyte.io/t/1148)] That will confirm which dependency is blocking startup. 3. Only then consider probe tuning The chart allows configuring liveness/readiness probe timings (initialDelaySeconds, timeout, etc.), as shown in the example snippet: [[Probe config](https://github.com/airbytehq/airbyte/issues/49843)] But changing probe settings is only useful if the server eventually becomes healthy. If it never binds to

:8001

because MinIO/Temporal/DB are unreachable, the probe will always fail and the container will keep being killed. --- So, the actionable path, grounded in the docs: - Fix MinIO’s Pending state by resolving its PVC/StorageClass issue. - Check `airbyte-server`’s

--previous

logs to confirm the dependency error. - Once dependencies are healthy, the liveness probe should pass and the restarts will stop.

Shawna Zhao

12/30/2025, 6:22 AM

@kapa.ai airbyte-minio-0 is from the previous deployment not relevent

kapa.ai

12/30/2025, 6:22 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:22 AM

Got it, thanks for clarifying that

airbyte-minio-0

is from a previous deployment and not relevant to the current one. Given the knowledge sources, I don’t see a documented case that exactly matches “server 2.0.1 on GKE, liveness probe failing with connection refused, no clear dependency issue, and an old MinIO pod from a previous release still around”. So I can’t point to a specific, confirmed root cause from the docs, but I can suggest next steps that are consistent with how similar issues were debugged. From the sources, when the server’s liveness probe fails with: >

Get "http://<pod-ip>:8001/api/v1/health": dial tcp ...: connect: connection refused

it always means the server process is not listening on

:8001

(either it hasn’t started yet or it failed during startup). [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] The underlying causes in documented cases were: - Failure to connect to Temporal. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655)] - Failure to connect to MinIO/S3. [[Minio logs](https://discuss.airbyte.io/t/1148)] - General platform misconfiguration (e.g., DNS/CoreDNS issues). [[Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655)] To move forward in a way that’s aligned with the docs: 1. Check the last crashed server logs Because the container is being killed by liveness, use `--previous`:

Copy code

bash kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test --previous

In other Kubernetes installs, this revealed explicit errors like: -

finishConnect(..) failed: Connection refused: airbyte-temporal...:7233

[[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655)] - S3/MinIO connection errors. [[Minio logs](https://discuss.airbyte.io/t/1148)] Without seeing that output, I can’t say which dependency is failing in your case. 2. Confirm current dependencies are healthy For the current release you just deployed (ignoring the old MinIO pod): - Check Temporal and DB pods are

Running

and look at their logs if the server log mentions them. [[K8s check conn](https://discuss.airbyte.io/t/594); [EKS timeout](https://discuss.airbyte.io/t/3127)] - If you are using MinIO or external S3 for logs/workload storage in this deployment, verify that configuration matches the docs and that the storage endpoint is reachable. [[Minio logs](https://discuss.airbyte.io/t/1148); [Workload storage liveness](https://github.com/airbytehq/airbyte/issues/57011)] 3. Clean up old resources if needed The docs mention that when reinstalling on Kubernetes, it’s valid to delete all resources in the Airbyte namespace (deployments, services, PVCs, etc.) to get a clean state. [[K8s check conn](https://discuss.airbyte.io/t/594)] If the old

airbyte-minio-0

is truly from a previous deployment and not referenced by the current Helm release, you can safely delete it and any associated PVCs, to avoid confusion:

Copy code

bash kubectl delete pod airbyte-minio-0 -n airbyte-test

(Only do this if you’re sure it’s not part of the current release.) --- Because the knowledge sources don’t contain an example of this exact GKE + 2.0.1 situation, I can’t give a definitive fix. The key next step, grounded in the docs, is to inspect

kubectl logs ... --previous

for

airbyte-server

and see which dependency it’s failing to connect to; that’s how similar issues with Temporal and MinIO were resolved. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server-temporal error](https://github.com/airbytehq/airbyte/issues/68655); [Minio logs](https://discuss.airbyte.io/t/1148)]

Shawna Zhao

12/30/2025, 6:24 AM

@kapa.ai kubectl logs ... --previous doesnt seem to show me the previous log

kapa.ai

12/30/2025, 6:24 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:25 AM

The knowledge sources don’t describe a case where

kubectl logs --previous

returns nothing for

airbyte-server

, so I can’t say exactly why it’s empty in your cluster. I can only suggest checks that are consistent with how Airbyte on Kubernetes is normally debugged. Given what we do have documented: 1. Confirm the container is actually restarting

--previous

only works if the container has restarted at least once. From your events you saw:

Copy code

text Container airbyte-server-container failed liveness probe, will be restarted

To double‑check, run:

Copy code

bash kubectl get pod airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test -o wide kubectl describe pod airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

Look for: -

Restart Count

> 0 for

airbyte-server-container

. - Under

Events

, repeated

Killing

Started

lines for that container. If

Restart Count

is still 0,

--previous

will not show anything. 2. Use plain
kubectl logs
right after a restart In other Airbyte K8s issues, server/worker logs were captured with a simple

kubectl logs

on the current container, not

--previous

. [[K8s logs](https://docs.airbyte.com/platform/1.6/contributing-to-airbyte/developing-locally#prerequisites)] Try:

Copy code

bash # Immediately after you see a "Killing ... failed liveness probe" event kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

If the container is starting and then being killed by the probe, there should be at least some startup logs before it dies (e.g., DB/Temporal connection attempts), as seen in other deployments. [[K8s check conn](https://discuss.airbyte.io/t/594); [EKS timeout](https://discuss.airbyte.io/t/3127)] 3. If logs are still empty, rely on
describe
+ events When logs aren’t available, the documented way to proceed on Kubernetes is to lean on

kubectl describe

and events to see why the pod is unhealthy. [[EKS timeout](https://discuss.airbyte.io/t/3127)] You already saw probe failures:

Copy code

text Readiness/Liveness probe failed: Get "http://<pod-ip>:8001/api/v1/health": dial tcp ...: connect: connection refused

From the knowledge sources, that always means the server process is not listening on

:8001

(never started correctly or is exiting very early). [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] Without logs, the only Airbyte‑specific guidance I can give (based on similar issues) is: - Verify Temporal and DB pods are

Running

and check their logs for errors, since server startup failures are often due to inability to connect to Temporal or the DB. [[K8s check conn](https://discuss.airbyte.io/t/594); [EKS timeout](https://discuss.airbyte.io/t/3127)] - Ensure there are no obvious cluster‑level issues (DNS, network) preventing the server from reaching those services, as seen in other K8s/GKE threads. [[K8s check conn](https://discuss.airbyte.io/t/594)] --- Because the sources don’t cover a case where

airbyte-server

has no logs at all and

--previous

is empty, I can’t provide a more specific, documented fix. The best next steps, aligned with the docs, are: 1. Use

kubectl describe pod

and events to confirm restarts and probe failures. 2. Try

kubectl logs

(without

--previous

) immediately after a restart. 3. Check Temporal and DB pods/logs for connectivity issues that could block server startup.

Shawna Zhao

12/30/2025, 6:28 AM

i dont see any error the pod just gets killed

Shawna Zhao

12/30/2025, 6:28 AM

@kapa.ai i dont see any error the pod just gets killed

kapa.ai

12/30/2025, 6:28 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:29 AM

I’m sorry, but the knowledge sources don’t contain an example where: -

airbyte-server

on Kubernetes/GKE - shows no errors in logs - and is only being killed by liveness/readiness probes with

connection refused

/api/v1/health

without any underlying startup or dependency error visible. In all documented cases that look similar, there is an underlying error in the server or platform: - Server fails to connect to Temporal and crashes or never becomes healthy. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server–Temporal](https://github.com/airbytehq/airbyte/issues/68655)] - Server fails to connect to MinIO/S3 or storage, causing platform issues. [[Minio logs](https://discuss.airbyte.io/t/1148); [Workload storage liveness](https://github.com/airbytehq/airbyte/issues/57011)] - Cluster‑level issues (DNS/CoreDNS, networking) prevent connections to services. [[Server–Temporal](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] In those cases, the sequence is: 1. Server starts, hits a dependency error (Temporal/DB/MinIO). 2. It never binds successfully to

:8001

or exits early. 3. Liveness/readiness probes to

http://<pod-ip>:8001/api/v1/health

get

connection refused

. 4. Kubelet kills and restarts the container. [[Quickstart probe](https://github.com/airbytehq/airbyte/issues/50430)] So even if you “don’t see any error” in the short log snippet you have, the documented pattern is that there is always some underlying failure (usually connectivity to Temporal, DB, or storage) that prevents the server from becoming healthy. Given the limits of the docs, the only Airbyte‑specific, evidence‑based steps I can suggest are: 1. Double‑check server logs around startup Run this a few seconds after a restart:

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

In similar issues, the important errors were near the top of the log and mentioned Temporal (

:7233

), DB, or S3/MinIO. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server–Temporal](https://github.com/airbytehq/airbyte/issues/68655); [Minio logs](https://discuss.airbyte.io/t/1148)] 2. Check Temporal and DB pods/logs Because documented failures often come from these:

Copy code

bash    kubectl get pods -n airbyte-test    kubectl logs <temporal-pod> -n airbyte-test    kubectl logs <db-pod> -n airbyte-test

Look for connection refused / DNS errors or crash loops. [[K8s check conn](https://discuss.airbyte.io/t/594); [Server–Temporal](https://github.com/airbytehq/airbyte/issues/68655)] 3. Use
kubectl describe pod
for
airbyte-server

Copy code

bash    kubectl describe pod airbyte-server-c6f5d5d4c-hvc2s -n airbyte-test

The events section is what we have in the docs for similar debugging on Kubernetes (e.g., for DB/Temporal connectivity or storage issues). [[EKS timeout](https://discuss.airbyte.io/t/3127); [K8s check conn](https://discuss.airbyte.io/t/594)] If, after these checks, there truly are no errors in any logs and only probe failures, that situation is not covered in the current knowledge sources, so I can’t provide a documented root cause or fix.

Shawna Zhao

12/30/2025, 6:30 AM

@kapa.ai Get "https://10.128.15.216:10250/containerLogs/airbyte-test/airbyte-temporal-7fb46bd89d-4stn5/airbyte-temporal": dial tcp 10.128.15.21610250 connect: connection refused

kapa.ai

12/30/2025, 6:31 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:31 AM

That error is coming from the Kubernetes API/kubelet when Airbyte (or

kubectl

) tries to fetch container logs from the node:

Copy code

text Get "<https://10.128.15.216:10250/containerLogs/airbyte-test/airbyte-temporal-7fb46bd89d-4stn5/airbyte-temporal>": dial tcp 10.128.15.216:10250: connect: connection refused

The knowledge sources don’t mention this exact message, but they do show that when Airbyte can’t talk to Temporal, you’ll see

Connection refused

errors at the service level (e.g.

airbyte-temporal/10.x.x.x:7233

) and pods will roll trying to connect. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [Temporal binding issue](https://github.com/airbytehq/airbyte/issues/50948)] In your case, the error is not about Temporal itself; it’s about the kubelet port (10250) on the node being unreachable, so Kubernetes can’t stream logs for the

airbyte-temporal

container. That’s a cluster/node or networking issue, not an Airbyte‑specific one. From the Airbyte side, what you can do (aligned with how other Temporal issues were debugged): 1. Check if the Temporal pod is actually running and healthy

Copy code

bash    kubectl get pod airbyte-temporal-7fb46bd89d-4stn5 -n airbyte-test    kubectl describe pod airbyte-temporal-7fb46bd89d-4stn5 -n airbyte-test

In similar issues, other pods (server/worker/cron) were restarting because they couldn’t connect to Temporal. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] 2. **Try getting logs via the node that does work** Since the error is

connect: connection refused

10.128.15.216:10250

, that node’s kubelet/logging endpoint is not reachable. You’ll need to fix that at the cluster level (firewall, kubelet, node health). The Airbyte docs don’t cover how to fix kubelet/10250, so I can’t give a documented procedure here. 3. Check other pods for Temporal connection errors If Temporal is unhealthy, you’ll typically see errors like:

Copy code

text    finishConnect(..) failed: Connection refused: airbyte-temporal/10.x.x.x:7233

airbyte-server

airbyte-worker

airbyte-cron

, etc. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] Use:

Copy code

bash    kubectl logs <airbyte-server-pod> -n airbyte-test    kubectl logs <airbyte-worker-pod> -n airbyte-test

Because the knowledge sources don’t describe fixing kubelet/10250 connectivity, I can’t provide a specific remediation for that part. From Airbyte’s perspective, you should: - Ensure the Temporal pod is healthy and reachable by other Airbyte services. - Resolve the node/kubelet connectivity issue so

kubectl logs

(and any Airbyte tooling that reads logs via the API) can access container logs.

Shawna Zhao

12/30/2025, 6:32 AM

can see log for db but not temporal @kapa.ai

kapa.ai

12/30/2025, 6:32 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:32 AM

You’re hitting a Kubernetes / node issue rather than an Airbyte‑specific one. The error you shared earlier:

Copy code

text Get "<https://10.128.15.216:10250/containerLogs/airbyte-test/airbyte-temporal-...>": dial tcp 10.128.15.216:10250: connect: connection refused

means the API server (or

kubectl

) cannot reach the kubelet on that node to stream any container logs from it. That’s why you can see logs for

db

(on a different node) but not for

airbyte-temporal

(on this node). The Airbyte knowledge sources only show Temporal‑level connection errors (e.g.,

Connection refused: airbyte-temporal/10.x.x.x:7233

) when other pods can’t talk to Temporal, not kubelet/10250 issues. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] Because of that: - Airbyte itself has no way to “fix” this; it depends on Kubernetes being able to access container logs. - The fix is at the cluster level (node/kubelet/network/firewall), which is not covered in the Airbyte docs, so I can’t give a documented step‑by‑step remediation. What you can still do from the Airbyte side, based on the docs: 1. Check if Temporal is healthy from other pods Look for Temporal connection errors in server/worker logs:

Copy code

bash    kubectl logs <airbyte-server-pod> -n airbyte-test    kubectl logs <airbyte-worker-pod> -n airbyte-test

If they show errors like:

Copy code

text    finishConnect(..) failed: Connection refused: airbyte-temporal/10.x.x.x:7233

then Temporal itself is not reachable, which is a known cause of server/worker restarts. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check conn](https://discuss.airbyte.io/t/594)] 2. Use
kubectl describe
on the Temporal pod

Copy code

bash    kubectl describe pod airbyte-temporal-7fb46bd89d-4stn5 -n airbyte-test

This is how other Temporal issues were debugged when logs weren’t enough (checking container state, restarts, and events). [[K8s check conn](https://discuss.airbyte.io/t/594)] If Temporal looks healthy from other pods and only log access is broken, that’s a Kubernetes/node problem outside what the Airbyte knowledge sources cover.

Shawna Zhao

12/30/2025, 6:32 AM

emporal Warning NodeNotReady 4m27s (x2 over 15m) node-controller Node is not ready

Shawna Zhao

12/30/2025, 6:33 AM

@kapa.ai i see emporal Warning NodeNotReady 4m27s (x2 over 15m) node-controller Node is not ready

kapa.ai

12/30/2025, 6:33 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:33 AM

NodeNotReady

on the node running Temporal is a Kubernetes/node health problem, and the Airbyte knowledge sources don’t provide specific remediation steps for that condition itself. They do, however, show that when Temporal is unhealthy or unreachable, Airbyte server/worker/cron pods will keep restarting or failing health checks because they can’t connect to Temporal. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check connection](https://discuss.airbyte.io/t/594)] In your case: - The node hosting

airbyte-temporal-...

NotReady

. - That explains why: - You can’t get Temporal logs (

on that node is unreachable). - Airbyte server liveness/readiness probes fail (server likely can’t talk to Temporal or is stuck waiting for it). From the Airbyte side, the only documented, relevant guidance is: 1. Temporal must be healthy and reachable Other pods (server, worker, cron, workload launcher) will roll or fail if they can’t connect to Temporal. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [Temporal binding issue](https://github.com/airbytehq/airbyte/issues/50948)] 2. Fix the Kubernetes node issue first Since the node is

NotReady

, you’ll need to resolve that at the cluster level (GKE node health / networking / kubelet) so: - The node becomes

Ready

again. - Temporal can run stably on a healthy node. - The API server can reach

to stream logs. 3. Then re‑check Airbyte pods After the node is

Ready

and Temporal is running: - Confirm

airbyte-temporal-...

Running

and healthy. - Check

airbyte-server

and

airbyte-worker

logs for any remaining Temporal connection errors like:

Copy code

text      finishConnect(..) failed: Connection refused: airbyte-temporal/…:7233

[[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check connection](https://discuss.airbyte.io/t/594)] If the node remains

NotReady

, Airbyte cannot operate correctly because Temporal (and any other pods on that node) will be effectively unavailable. The Airbyte docs don’t cover how to repair

NodeNotReady

on GKE, so I can’t provide a documented fix for that part.

Shawna Zhao

12/30/2025, 6:34 AM

@kapa.ai how to fix the node not ready

kapa.ai

12/30/2025, 6:34 AM

A message from kapa.ai

kapa.ai

12/30/2025, 6:34 AM

The Airbyte knowledge sources you provided do not include instructions on how to fix a

NodeNotReady

condition in Kubernetes/GKE. They only show that when Temporal is on an unhealthy node, other Airbyte pods (server, worker, cron, workload launcher) fail because they can’t connect to Temporal. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check connection](https://discuss.airbyte.io/t/594)] Because of that, I can’t give you a documented, Airbyte‑specific procedure to fix

NodeNotReady

itself. What is clear from the docs is: - Airbyte requires Temporal to be healthy and reachable; if the node running Temporal is

NotReady

, Airbyte will not work correctly. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655)] - The

NodeNotReady

condition is a cluster‑level issue (GKE node / kubelet / networking / underlying VM), and needs to be resolved using Kubernetes/GKE tooling and documentation, which are outside the scope of the Airbyte docs. So, from an Airbyte perspective, the next step is: - Use GKE/Kubernetes documentation or your cluster admin tools to bring that node back to

Ready

(or cordon/drain and replace it). - Once the node is healthy and Temporal is running on a

Ready

node, re‑check: -

airbyte-temporal

pod is

Running

. -

airbyte-server

and

airbyte-worker

logs no longer show Temporal connection errors like

Connection refused: airbyte-temporal/...:7233

. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655); [K8s check connection](https://discuss.airbyte.io/t/594)] I’m limited to what’s in the Airbyte knowledge sources, and they don’t describe how to repair

NodeNotReady

itself.

Shawna Zhao

12/30/2025, 11:00 AM

@kapa.ai i kicked off the fresh install again. three pods keeps restarting airbyte-server-c6f5d5d4c-cjd9b,airbyte-worker-5ccc858c87-gx2tw,airbyte-workload-launcher-667784c4b9-jlg74

kapa.ai

12/30/2025, 11:00 AM

A message from kapa.ai

kapa.ai

12/30/2025, 11:00 AM

The knowledge sources don’t describe your exact GKE + 2.x setup, but they do cover very similar symptoms: server, worker, and workload‑launcher all restarting because they can’t reach core dependencies (DB, Temporal, storage). [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [Workload launcher JDBC](https://github.com/airbytehq/airbyte/discussions/55862)] Given that three core pods are restarting together: -

airbyte-server

airbyte-worker

airbyte-workload-launcher

the documented root causes are almost always: 1. Database connectivity problems For example, in a recent Helm install,

airbyte-workload-launcher

and

airbyte-worker

were in back‑off with: >

sql handle: unable to refresh database connection pool

dial tcp <db-ip>:5432: connect: connection refused

[[Helm DB error](https://github.com/airbytehq/airbyte/issues/68136)] And in another case, workload‑launcher failed with: >

Failed to obtain JDBC Connection

when talking to the workload‑api DB. [[Workload launcher JDBC](https://github.com/airbytehq/airbyte/discussions/55862)] What to check:

Copy code

bash    kubectl logs airbyte-workload-launcher-667784c4b9-jlg74 -n <ns>    kubectl logs airbyte-worker-5ccc858c87-gx2tw -n <ns>    kubectl logs airbyte-server-c6f5d5d4c-cjd9b -n <ns>

Look specifically for: -

Failed to obtain JDBC Connection

connect: connection refused

to port

- Any

sql handle: unable to refresh database connection pool

messages If you see those, the DB service/endpoint or credentials are wrong/unreachable. 2. Temporal not reachable When Temporal is misconfigured or unreachable, server/worker/cron/launcher all roll trying to connect. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [K8s check connection](https://discuss.airbyte.io/t/594)] What to check: - Confirm Temporal pod is `Running`:

Copy code

bash      kubectl get pod -n <ns> | grep temporal

- Look in server/worker logs for errors like:

Copy code

text      finishConnect(..) failed: Connection refused: airbyte-temporal-svc/…:7233

[[K8s check connection](https://discuss.airbyte.io/t/594); [Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948)] 3. Storage / workload‑output issues (workload‑launcher) There is a known issue where

airbyte-workload-launcher

can get into a bad state when it loses access to the configured storage (MinIO/S3), and liveness probes don’t catch it well. [[Workload liveness](https://github.com/airbytehq/airbyte/issues/57011)] In your case, since it’s restarting (not just stuck), check its logs for: - Errors writing logs or workload output to S3/MinIO. - Any stack traces mentioning storage clients. --- ### Concrete next steps (based on the docs) 1. Get logs from the three restarting pods

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-cjd9b -n <ns>    kubectl logs airbyte-worker-5ccc858c87-gx2tw -n <ns>    kubectl logs airbyte-workload-launcher-667784c4b9-jlg74 -n <ns>

If they are crash‑looping, also try `--previous`:

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-cjd9b -n <ns> --previous

Compare what you see with the documented patterns: - DB connection refused / JDBC errors. [[Helm DB error](https://github.com/airbytehq/airbyte/issues/68136); [Workload launcher JDBC](https://github.com/airbytehq/airbyte/discussions/55862)] - Temporal connection refused on

:7233

. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [K8s check connection](https://discuss.airbyte.io/t/594)] - Storage / S3 / MinIO errors. [[Workload liveness](https://github.com/airbytehq/airbyte/issues/57011)] 2. Describe one of the pods to confirm restart reason

Copy code

bash    kubectl describe pod airbyte-server-c6f5d5d4c-cjd9b -n <ns>

Look for: -

Last State: Terminated

with a non‑zero exit code. - Events mentioning

Back-off restarting failed container

or probe failures. 3. If logs show DB errors - Verify your DB host, port, user, password, and that the DB is reachable from the cluster (security groups / firewall). - This is exactly what caused the Helm install failures in the referenced issue. [[Helm DB error](https://github.com/airbytehq/airbyte/issues/68136); [Workload launcher JDBC](https://github.com/airbytehq/airbyte/discussions/55862)] 4. If logs show Temporal errors - Ensure the Temporal service DNS (

airbyte-temporal-svc

or similar) resolves and the pod is healthy. - Temporal connectivity issues are a known cause of server/worker/cron/launcher restarts. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [K8s check connection](https://discuss.airbyte.io/t/594)] --- If you can paste a few lines of error from any of those three pods, I can map it directly to the closest documented issue (DB vs Temporal vs storage) and suggest a more specific fix.

Shawna Zhao

12/30/2025, 11:04 AM

@kapa.ai error in the workload launcher

Copy code

ERROR	i.a.a.c.a.AccessTokenInterceptor(intercept):120 - Failed to obtain or add access token
java.net.ConnectException: Failed to connect to airbyte-airbyte-server-svc.airbyte-test/34.118.227.31:8001
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:297)
	at okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:207)
	at okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
	at okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
	at okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
	at okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
	at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at io.airbyte.api.client.auth.AccessTokenInterceptor.fetchNewToken(AccessTokenInterceptor.kt:96)
	at io.airbyte.api.client.auth.AccessTokenInterceptor.getCachedToken(AccessTokenInterceptor.kt:70)
	at io.airbyte.api.client.auth.AccessTokenInterceptor.intercept(AccessTokenInterceptor.kt:116)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at io.airbyte.api.client.interceptor.ThrowOn5xxInterceptor.intercept(ThrowOn5xxInterceptor.kt:14)
	at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
	at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
	at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
	at dev.failsafe.okhttp.FailsafeCall.lambda$execute$0(FailsafeCall.java:117)
	at dev.failsafe.Functions.lambda$get$0(Functions.java:46)
	at dev.failsafe.internal.RetryPolicyExecutor.lambda$apply$0(RetryPolicyExecutor.java:74)
	at dev.failsafe.SyncExecutionImpl.executeSync(SyncExecutionImpl.java:187)
	at dev.failsafe.CallImpl.execute(CallImpl.java:33)
	at dev.failsafe.okhttp.FailsafeCall.execute(FailsafeCall.java:121)
	at io.airbyte.api.client.generated.DataplaneApi.heartbeatDataplaneWithHttpInfo(DataplaneApi.kt:1897)
	at io.airbyte.api.client.generated.DataplaneApi.heartbeatDataplane(DataplaneApi.kt:293)
	at io.airbyte.api.client.generated.DataplaneApi.heartbeatDataplane$default(DataplaneApi.kt:290)
	at io.airbyte.workload.launcher.authn.DataplaneIdentityService.heartbeat(DataplaneIdentityService.kt:70)
	at io.airbyte.workload.launcher.authn.$DataplaneIdentityService$Definition$Exec.dispatch(Unknown Source)
	at io.micronaut.context.AbstractExecutableMethodsDefinition$DispatchedExecutableMethod.invoke(AbstractExecutableMethodsDefinition.java:456)
	at io.micronaut.inject.DelegatingExecutableMethod.invoke(DelegatingExecutableMethod.java:86)
	at io.micronaut.context.bind.DefaultExecutableBeanContextBinder$ContextBoundExecutable.invoke(DefaultExecutableBeanContextBinder.java:152)
	at io.micronaut.scheduling.processor.ScheduledMethodProcessor.lambda$scheduleTask$2(ScheduledMethodProcessor.java:160)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
	at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:358)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
	at java.base/java.lang.Thread.run(Thread.java:1583)
Caused by: java.net.ConnectException: Connection refused
	at java.base/sun.nio.ch.Net.pollConnect(Native Method)
	at java.base/sun.nio.ch.Net.pollConnectNow(Net.java:682)
	at java.base/sun.nio.ch.NioSocketImpl.timedFinishConnect(NioSocketImpl.java:542)
	at java.base/sun.nio.ch.NioSocketImpl.connect(NioSocketImpl.java:592)
	at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:327)
	at java.base/java.net.Socket.connect(Socket.java:751)
	at okhttp3.internal.platform.Platform.connectSocket(Platform.kt:128)
	at okhttp3.internal.connection.RealConnection.connectSocket(RealConnection.kt:295)
	... 44 common frames omitted

kapa.ai

12/30/2025, 11:05 AM

A message from kapa.ai

kapa.ai

12/30/2025, 11:05 AM

That stack trace matches a known pattern: the workload-launcher cannot reach the Airbyte server service, so it fails to obtain an access token and keeps restarting. The key line is:

Copy code

text Failed to connect to airbyte-airbyte-server-svc.airbyte-test/34.118.227.31:8001 Caused by: java.net.ConnectException: Connection refused

A very similar error is reported when other components try to call

airbyte-airbyte-server-svc:8001

and get

Connection refused

during Helm/Kubernetes installs. [[Custom source API](https://github.com/airbytehq/airbyte/issues/68608); [Chart 2.0.0 issues](https://github.com/airbytehq/airbyte/issues/68134)] In your case, this means: -

airbyte-workload-launcher

is trying to call the server (for dataplane heartbeat / access token). - The TCP connection to

airbyte-airbyte-server-svc:8001

is refused. - So the launcher logs

Failed to obtain or add access token

and restarts. To debug using what’s in the knowledge sources: 1. Check that the server pod is actually healthy and listening on 8001 - Get server pod status and events:

Copy code

bash      kubectl get pod -n airbyte-test | grep airbyte-server      kubectl describe pod airbyte-server-c6f5d5d4c-cjd9b -n airbyte-test

- In similar issues, the server was failing liveness/readiness probes on

:8001/api/v1/health

with

connection refused

, so it kept restarting and was never reachable. [[K8s check connection](https://discuss.airbyte.io/t/594); [Chart 2.0.0 issues](https://github.com/airbytehq/airbyte/issues/68134)] 2. Check server logs for why it’s not accepting connections

Copy code

bash    kubectl logs airbyte-server-c6f5d5d4c-cjd9b -n airbyte-test    kubectl logs airbyte-server-c6f5d5d4c-cjd9b -n airbyte-test --previous

In other installs, the server failed to start because it couldn’t connect to Temporal or other dependencies, with errors like:

Copy code

text    finishConnect(..) failed: Connection refused: airbyte-temporal/10.x.x.x:7233

which caused all server calls to fail. [[Temporal bind issue](https://github.com/airbytehq/airbyte/issues/50948); [Server–Temporal error](https://github.com/airbytehq/airbyte/issues/68655)] 3. Verify the server service resolves and routes correctly From another pod (e.g. worker):

Copy code

bash    kubectl exec -it airbyte-worker-5ccc858c87-gx2tw -n airbyte-test -- sh -c \      'apk add --no-cache curl >/dev/null 2>&1 || true; \       getent hosts airbyte-airbyte-server-svc.airbyte-test; \       curl -v <http://airbyte-airbyte-server-svc.airbyte-test:8001/api/v1/health>'

In the similar issue about custom source definitions, the error was exactly:

Copy code

text    Failed to connect to airbyte-airbyte-server-svc/172.20.113.6:8001

indicating the service IP was reachable at the network level but nothing was listening (server not healthy). [[Custom source API](https://github.com/airbytehq/airbyte/issues/68608)] 4. Once the server is healthy, the launcher error should disappear The launcher’s stack trace shows it’s failing during

DataplaneApi.heartbeatDataplane

, which depends on the server being up and able to issue tokens. When the server is reachable, this call succeeds and the

Failed to obtain or add access token

error goes away. [[Custom source API](https://github.com/airbytehq/airbyte/issues/68608); [Chart 2.0.0 issues](https://github.com/airbytehq/airbyte/issues/68134)] --- So the actionable path, based on the docs: - Focus on why
airbyte-airbyte-server-svc:8001
is refusing connections (server pod not healthy / not listening). - Use

kubectl describe

and

kubectl logs

on the server pod to find dependency errors (especially Temporal). - After the server is stable and

/api/v1/health

returns

UP

from inside the cluster, the workload-launcher should stop logging this access-token connection error.

kapa.ai

12/30/2025, 11:09 AM

A message from kapa.ai

kapa.ai

12/30/2025, 11:09 AM

This error is coming from Kubernetes itself, not from Airbyte:

Copy code

text Error from server: Get "<https://10.128.15.227:10250/containerLogs/...>": dial tcp 10.128.15.227:10250: connect: connection refused

It means the API server (or

kubectl

) cannot reach the kubelet on the node

10.128.15.227

to stream container logs. The Airbyte knowledge sources only show application‑level “connection refused” issues (e.g., server ↔️ Temporal, server ↔️ MinIO), not kubelet/10250 failures, and they do not document how to fix this kind of node/kubelet problem. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [K8s check connection](https://discuss.airbyte.io/t/594)] From the Airbyte side, the implications are: - Any Airbyte pod scheduled on that node (e.g. your

airbyte-workload-launcher

pod) will have unreadable logs via

kubectl logs

, because the API server can’t talk to that node’s kubelet. - Airbyte itself cannot change or fix this; it depends on a healthy Kubernetes cluster. The Airbyte docs do not provide steps to repair: - kubelet on port

, or - a

NodeNotReady

/ unreachable node situation. So I can’t give a documented procedure to restore

kubectl logs

for that node. What is consistent with the knowledge sources: - Airbyte components (server, worker, workload‑launcher, Temporal) must run on healthy nodes and be able to reach each other (DB, Temporal, storage). When a node is not ready or unreachable, pods on it will misbehave and other components may show connection‑refused errors. [[Server–Temporal error](https://github.com/airbytehq/airbyte/issues/50948); [K8s check connection](https://discuss.airbyte.io/t/594)] Given the limits of the docs, the next steps are: - Use your GKE/Kubernetes tooling and documentation to fix the node/kubelet issue on

10.128.15.227

(or cordon/drain and replace that node). - Once the node is healthy and reachable on

kubectl logs

for the Airbyte pods on that node should start working again.

2 Views

Open in Slack

Previous Next