@kapa.ai i have this in my job, and all my airbyte pods restarted where should i start trouble shoot

026-05-06 05:14:37 replication-orchestrator WARN Retry attempt 1 of 5. Last response: null
2026-05-06 05:14:37 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:37 replication-orchestrator ERROR DestinationReader error: 
2026-05-06 05:14:37 replication-orchestrator INFO DestinationReader finished.
2026-05-06 05:14:37 replication-orchestrator ERROR runJobs failed; recording failure but continuing to finish.
2026-05-06 05:14:37 replication-orchestrator INFO Closing StateCheckSumCountEventHandler
2026-05-06 05:14:37 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:38 replication-orchestrator WARN Retry attempt 2 of 5. Last response: null
2026-05-06 05:14:38 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:39 replication-orchestrator WARN Retry attempt 1 of 5. Last response: null
2026-05-06 05:14:39 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:40 replication-orchestrator WARN Retry attempt 3 of 5. Last response: null
2026-05-06 05:14:40 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:41 replication-orchestrator WARN Retry attempt 2 of 5. Last response: null
2026-05-06 05:14:41 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:42 replication-orchestrator WARN Retry attempt 4 of 5. Last response: null
2026-05-06 05:14:42 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:43 replication-orchestrator WARN Retry attempt 3 of 5. Last response: null
2026-05-06 05:14:43 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:44 replication-orchestrator WARN Retry attempt 5 of 5. Last response: null
2026-05-06 05:14:44 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:44 replication-orchestrator ERROR Retry attempts exceeded.
2026-05-06 05:14:44 replication-orchestrator ERROR Log message invocation failed: java.lang.NullPointerException: Cannot invoke "retrofit2.Response.raw()" because the return value of "dev.failsafe.event.ExecutionCompletedEvent.getResult()" is null
2026-05-06 05:14:44 replication-orchestrator WARN Error sending heartbeat
2026-05-06 05:14:45 replication-orchestrator WARN Retry attempt 4 of 5. Last response: null
2026-05-06 05:14:45 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:47 replication-orchestrator WARN Retry attempt 5 of 5. Last response: null
2026-05-06 05:14:47 replication-orchestrator ERROR Failed to obtain or add access token
2026-05-06 05:14:47 replication-orchestrator ERROR Retry attempts exceeded.
2026-05-06 05:14:47 replication-orchestrator ERROR Failed to call unknown.  Last response: null

The errors you're seeing —

Failed to obtain or add access token

with

Last response: null

— indicate that your replication-orchestrator pods cannot authenticate with the Airbyte platform services (specifically the dataplane/workload API). Since all your pods restarted, this is likely the root cause. Troubleshooting Steps 1. Check the airbyte-auth-secrets Secret This is the most commonly reported root cause. The

airbyte-auth-secrets

Kubernetes secret must exist and contain valid dataplane client credentials. [GitHub Issue]

kubectl get secret airbyte-auth-secrets -n <your-namespace>
kubectl describe secret airbyte-auth-secrets -n <your-namespace>

If this secret was pre-created and managed externally, it may not be getting updated correctly even if you have the right RBAC roles configured. One reported finding is that externally managed secrets can prevent the platform from updating the credentials properly. [GitHub Issue] 2. Verify the Workload API Server is Reachable After a pod restart, the workload API server may not be fully up yet. Check if it's running and reachable:

kubectl get pods -n <your-namespace> | grep workload-api-server
kubectl logs <workload-api-server-pod> -n <your-namespace>

A

ConnectException: Failed to connect to airbyte-helm-workload-api-server-svc

is a known symptom when the workload API server pod is down or not yet ready. [GitHub Issue] 3. Check for Token Request Failures (HTTP 500) The

Last response: null

suggests the token endpoint may be returning an error or is unreachable. Look at the server-side logs for the auth/dataplane service:

kubectl logs <airbyte-server-pod> -n <your-namespace> | grep -i "token\|auth\|500"

A

Token request failed with HTTP code 500

has been reported in similar scenarios. [GitHub Issue] 4. Check Workload API Authentication Config (Kubernetes/Helm) If you're on Kubernetes with Helm, verify that

API_AUTHORIZATION_ENABLED

,

WORKLOAD_API_AUTH_HEADER

, and

WORKLOAD_API_BEARER_TOKEN

are properly set in your ConfigMap and that these values are propagating to the connector/orchestrator containers. [GitHub Issue] 5. Wait for All Pods to Stabilize Since all pods restarted, some services may still be initializing. Give the cluster a few minutes and then check pod status:

kubectl get pods -n <your-namespace>

Once all pods are in

Running

state, retry the sync to see if the issue resolves itself. Summary: The most likely causes are (a) the

airbyte-auth-secrets

secret is missing or stale after the pod restarts, or (b) the workload API server is not yet reachable. Start with checking the secret and pod health. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

my airbyte server had a restart

Given your previous context (the

Failed to obtain or add access token

errors after a pod restart), an Airbyte server restart is a known trigger for this type of authentication failure. Here's what to focus on: After an Airbyte Server Restart 1. Verify the airbyte-auth-secrets Secret is Intact The server restart may have caused issues with the

airbyte-auth-secrets

secret. Check that it exists and contains the expected keys:

kubectl get secret airbyte-auth-secrets -n <your-namespace> -o yaml

You should see keys like

instance-admin-client-id

,

instance-admin-client-secret

, and

instance-admin-password

. [Authentication Docs] 2. Let Airbyte Manage the Secret Itself A key finding from community reports: if the

airbyte-auth-secrets

secret is pre-created or externally managed, the server may not be able to update it with the correct dataplane credentials after a restart. The recommended fix is to: • Remove the external management of the secret • Delete the existing secret from the cluster • Let Airbyte recreate it on startup (it will generate the

dataplane-client-id

and

dataplane-client-secret

automatically) [GitHub Issue] 3. Confirm All Pods Are Healthy

kubectl get pods -n <your-namespace>

Make sure the

airbyte-server

,

workload-api-server

, and

workload-launcher

pods are all in

Running

state before retrying syncs. [GitHub Issue] 4. Retry Your Sync Once all pods are stable, retry the failing sync. The token errors are often transient and resolve once the server has fully initialized and recreated the necessary credentials. If the issue persists after all pods are running, the

airbyte-auth-secrets

secret management is the most likely culprit based on reported cases. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what is the common reason for airbyte server to have a restart

Based on the knowledge sources available, here are the documented common reasons for Airbyte server/pod restarts: Common Reasons for Airbyte Pod Restarts 1. Out-of-Memory (OOM) Issues • Resource leak in the worker: The

airbyte-worker

process can accumulate thousands of threads over time (e.g., due to a log4j2 S3 appender bug), consuming excessive RAM and eventually causing the pod to become unresponsive or be killed. [Forum Post] • OOM on newer Linux kernels (6.12+): On kernel 6.12 and above, OpenJDK can no longer detect the cpuset cgroup and falls back to host memory, causing it to set a heap far above the pod's actual limit — leading to OOM kills. Downgrading the kernel (e.g., from 6.14 to 6.8) has been the reported fix. [GitHub Issue] • Insufficient resource limits: If job container memory limits are too low (e.g., defaulting to 256Mi), pods can be OOMKilled. [GitHub Issue] 2. Kubernetes Cluster Autoscaler Evictions On GKE or similar managed Kubernetes clusters, the cluster autoscaler can evict critical pods like

temporal

and

workload-api-server

during scale-down events, causing cascading failures and restarts. The workaround is to annotate those pods as safe-to-evict=false. [GitHub Issue] 3. Database Connectivity Issues If the underlying PostgreSQL database is unreachable (e.g., connection refused), pods like

temporal

and

workload-launcher

will crash-loop and restart. [GitHub Issue] 4. Authentication/Secret Misconfiguration Issues with the

airbyte-auth-secrets

Kubernetes secret (e.g., missing keys like

dataplane-client-id

) can cause pods to fail to start or crash, triggering restarts. [GitHub Issue] 5. Insufficient Host Resources Running Airbyte on an under-provisioned instance (e.g., too little RAM, heavy swapping) can cause services to become unresponsive and restart. [Forum Post] In your specific case, given the

Failed to obtain or add access token

errors, the restart was most likely triggered by an authentication/secret issue or a transient infrastructure event (like an autoscaler eviction or OOM), rather than a connector-level problem. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.