https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Our Airbyte servers have been sudde...
# replication-ask-ai
a
@kapa.ai Our Airbyte servers have been suddenly, and randomly failing. We have 30 connections set up between Postgres & Google BigQuery, and run manual syncs. One day, all of our syncs got stuck in "Queued" forever, and never ran. Trying to "refresh source schema" led to these errors in the console:
Failed to load resource: the server responded with a status of 500 (/api/v1/web_backend/connections/get:1 )
Understand this error
status:1 Failed to load resource: the server responded with a status of 401 ()Understand this error
core-o3gxm74u91.js:121 🔁 Retrying request to /api/v1/connections/status due to unauthorized (HTTP 401). Retry 1/3
latest_per_run_state:1 Failed to load resource: the server responded with a status of 401 ()Understand this error
core-o3gxm74u91.js:121 🔁 Retrying request to /api/v1/stream_statuses/latest_per_run_state due to unauthorized (HTTP 401). Retry 1/3
Now, all 30 connections, which were working literally yesterday, and suddenly all broken and cannot connect. Help me troubleshoot this error. We are using version 1.6.2, the free version of AirByte
k
It looks like your issue combines two separate symptoms that we partially recognize from the knowledge sources, but there isn’t a single documented root cause for exactly your scenario (v1.6.2, free/self‑managed, all jobs stuck in “Queued” + 401s on 
/connections/status
). I’ll outline what we can say from the docs and similar reports, and concrete next steps to debug. 1. “Queued forever” – what it usually means In Airbyte Cloud, “Queued” explicitly means “waiting for worker capacity.” [Connection status] On self‑managed, similar symptoms (jobs stuck in 
Queued
or 
Starting
and never progressing) have been seen when: • The worker / workload components can’t actually start jobs (e.g., worker pod stuck in 
Pending
, or internal API auth between server and worker is broken). [All sync jobs hanging; Syncs getting stuck] • The deployment is unhealthy (Temporal / scheduler / worker not reachable). [Kubernetes check connection issues; Temporal scheduler causing hangs] In those cases, the UI shows jobs as “Queued” or “Running”, but nothing actually executes. Action items: If you’re on Kubernetes: 1. Check pod health in the Airbyte namespace:kubectl get pods -n your-airbyte-namespacePay special attention to:If worker is 
Pending
or CrashLooping, that explains the “Queued forever” behavior. [Syncs getting stuck] ◦ 
airbyte-worker
/ 
airbyte-abctl-worker
◦ 
airbyte-server
/ 
airbyte-abctl-server
◦ 
airbyte-abctl-workload-api-server
◦ Temporal / scheduler pods (if present) 1. If you see 401s related to the Workload API in logs, there is a known pattern where the WORKLOAD_API_BEARER_TOKEN / auth header is inconsistent between server, worker, and workload‑api‑server, causing all workload API calls to be rejected with 401 and preventing jobs from starting. [Workload 401 fix]The workaround from that issue is to set the same token on all three deployments and restart them, for example:TOKEN=$(openssl rand -hex 24) kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl set env deployment/airbyte-abctl-server \ WORKLOAD_API_AUTH_HEADER=Authorization \ WORKLOAD_API_BEARER_TOKEN=$TOKEN kubectl -n airbyte-abctl rollout restart deploy/airbyte-abctl-workload-api-server airbyte-abctl-worker airbyte-abctl-serverOr the variant that also sets 
INTERNAL_API_AUTH_TOKEN
on the worker: [Workload 401 fix]BEARER_TOKEN=$(openssl rand -hex 16) kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \ WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN" \ INTERNAL_API_AUTH_TOKEN="$BEARER_TOKEN" kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \ WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN" kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server If you’re on Docker: • The knowledge sources don’t give a direct analogue for 1.6.x + Docker, but similar “everything stuck” situations in older Docker installs were often due toSo ◦ Airbyte DB / logs filling disk so containers crash or become unresponsive. [Syncs and tests failing suddenly; Long loading time & 502] ◦ Memory leaks in logging causing the server to become extremely slow or return 502s until containers are restarted. [Long loading time & 502] ◦ Check 
docker ps
and 
docker logs
for 
airbyte-server
and 
airbyte-worker
. ◦ Check disk space and container memory usage (
docker stats
). ◦ If disk is full (e.g., “No space left on device”), you may need to prune logs or old job history and then restart containers. [Syncs and tests failing suddenly] 2. 401s from /api/v1/connections/status and /stream_statuses/latest_per_run_state The browser console shows: • 500 on 
/api/v1/web_backend/connections/get
• 401 on 
/api/v1/connections/status
• 401 on 
/api/v1/stream_statuses/latest_per_run_state
• The frontend retries 401s a few times (this matches the documented pattern: on 401, retry once after refreshing the token). [API token expiry handling] From the docs: • Access tokens are short‑lived; on 401 you should request a new token and retry once. [API token expiry handling] In your case, the UI is already doing that, but still getting 401s. That usually means: • The backend is rejecting all requests as unauthorized (e.g., misconfigured auth / tokens between components, as in the Workload API 401 issue above). [Workload 401 fix] or • The server is in a bad state (e.g., memory / DB / internal services) and returning 500/401 inconsistently, which we’ve seen in older versions manifest as 502/500s and very slow UI. [Long loading time & 502; Unable to access webapp, 502s] Action items: 1. Check 
airbyte-server
logs around the time you see the 500/401 in the browser. Look for: ◦ Any 
Unauthorized
/ 401 messages ◦ Errors mentioning “workload” or “workload api” ◦ DB / disk / out‑of‑memory errors 1. If you’re on Kubernetes with abctl/Helm and see 401s related to workload API, apply the token‑alignment workaround above. [Workload 401 fix] 2. If logs show DB / disk / memory issues, free space or resources and restart 
airbyte-server
and 
airbyte-worker
. This has been the recommended workaround for similar “UI unusable / 5xx / long load” issues. [Syncs and tests failing suddenly; Long loading time & 502] 3. Version considerations You’re on 1.6.2. The 1.6.x patch notes mention: • 1.6.1 fixed some install/upgrade failures with abctl/Helm. • 1.6.2 fixed a generic OIDC bug. • Later 1.6.x releases are mostly auth and security fixes. [Airbyte 1.6 patches] The knowledge sources don’t explicitly tie your exact symptoms to a specific 1.6.2 bug, but: • If you’re using generic OIDC, 1.6.2 was specifically to fix an auth issue. • If you’re on abctl/Helm and recently upgraded, double‑check that the upgrade completed cleanly and that all pods are on the same version. 4. What would help narrow this down Because the knowledge sources don’t describe this exact combination (1.6.2 + free + all connections suddenly 401/Queued), the next best step is to gather a bit more: • Deployment type: Docker vs Kubernetes (Helm/abctl) and which namespace. • Output of: ◦ 
kubectl get pods -n <namespace>
(or 
docker ps
) ◦ Any 401/500 stack traces from 
airbyte-server
logs ◦ Whether you see “Workload API” / 
WORKLOAD_API_BEARER_TOKEN
/ 
INTERNAL_API_AUTH_TOKEN
mentioned in logs. [Workload 401 fix] If you can share those details (sanitized), we can map them more directly to the known 401/Queued patterns in the references above and suggest a more precise fix. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai docker stats returns airbyte-abctl-control-plane, cpu % ~20%, Mem % 44.38% and net i/o at 35.9GB/25.3GB, block i/o at ~20% kubectl is not returning anything useful, and I'm not sure what the namespace would be called
k
Given the knowledge sources, I can’t see an exact match for your Docker‑only + abctl setup symptoms, so I’ll stay close to what is documented and avoid guessing. From the issues we do have, two things stand out: 1. abctl uses a Kind cluster inside the 
airbyte-abctl-control-plane
container, and the Airbyte namespace there is typically 
airbyte-abctl
. In multiple reports, users had to: ◦ 
docker exec
into the 
airbyte-abctl-control-plane
container ◦ then run 
kubectl -n airbyte-abctl get pods
from inside that container. [1.6.0 dataplane issue; Workload 401 fix] 1. Random 401s / 5xx and jobs stuck are often tied to misaligned internal auth tokens between worker, server, and workload‑api‑server in abctl/Helm installs. Aligning 
WORKLOAD_API_BEARER_TOKEN
(and sometimes 
INTERNAL_API_AUTH_TOKEN
) across those deployments and restarting them has resolved 502/500/401 and “connections not working” for others. [Workload 401 fix; 502 during connector config] Given that: 1. Get real kubectl output (inside the control‑plane container) Right now 
kubectl
“not returning anything useful” is likely because you’re running it on the host, not inside the Kind cluster that abctl created. Try:
Copy code
# 1. Find the control-plane container (you already have its name)
docker ps | grep airbyte-abctl-control-plane

# 2. Exec into it
docker exec -it airbyte-abctl-control-plane bash

# 3. Inside the container, list pods in the abctl namespace
kubectl -n airbyte-abctl get pods
You should see pods like: • 
airbyte-abctl-server-...
• 
airbyte-abctl-worker-...
• 
airbyte-abctl-workload-api-server-...
• 
airbyte-abctl-webapp-...
• etc. [1.6.0 dataplane issue] If any of these are 
CrashLoopBackOff
/ 
Error
/ not 
Running
, that’s a strong signal why jobs are stuck and APIs are failing. 2. Check for the known 401 / workload‑API auth issue Inside the same shell (still in 
airbyte-abctl-control-plane
):
Copy code
kubectl -n airbyte-abctl logs deployment/airbyte-abctl-worker | grep -i "unauthorized\|401\|workload"
kubectl -n airbyte-abctl logs deployment/airbyte-abctl-workload-api-server | grep -i "unauthorized\|401\|workload"
kubectl -n airbyte-abctl logs deployment/airbyte-abctl-server | grep -i "unauthorized\|401\|workload"
In a very similar case, users saw 401s because 
WORKLOAD_API_BEARER_TOKEN
(and header) were not aligned between worker and workload‑api‑server, causing workload API calls to fail and connector operations to break. [Workload 401 fix] If that matches what you see, apply the documented workaround:
Copy code
# still inside airbyte-abctl-control-plane

TOKEN=$(openssl rand -hex 24)

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \
  WORKLOAD_API_AUTH_HEADER=Authorization \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \
  WORKLOAD_API_AUTH_HEADER=Authorization \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-server \
  WORKLOAD_API_AUTH_HEADER=Authorization \
  WORKLOAD_API_BEARER_TOKEN=$TOKEN

kubectl -n airbyte-abctl rollout restart deploy/airbyte-abctl-workload-api-server airbyte-abctl-worker airbyte-abctl-server
Or the variant that also sets 
INTERNAL_API_AUTH_TOKEN
on the worker (this is what another user used to fix 502s when configuring connectors): [Workload 401 fix; 502 during connector config]
Copy code
BEARER_TOKEN=$(openssl rand -hex 16)
echo "Generated token: $BEARER_TOKEN"

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-worker \
  WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN" \
  INTERNAL_API_AUTH_TOKEN="$BEARER_TOKEN"

kubectl -n airbyte-abctl set env deployment/airbyte-abctl-workload-api-server \
  WORKLOAD_API_BEARER_TOKEN="$BEARER_TOKEN"

kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-worker
kubectl -n airbyte-abctl rollout restart deployment airbyte-abctl-workload-api-server

kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-worker
kubectl -n airbyte-abctl rollout status deployment airbyte-abctl-workload-api-server
This exact sequence is what other abctl users on 1.6.x / 1.8.x used to resolve “Airbyte temporarily unavailable / HTTP 502” and connector test failures. [502 during connector config; Workload 401 fix] 3. If things are still broken The knowledge sources also show that: • Some users with abctl‑based installs that got into a bad state simply did 
abctl local uninstall
then 
abctl local install
to recover. [Unending helm debug / reinstall] • Others hit issues in 1.6.x around dataplane / auth secrets and sometimes rolled back or moved to a later patch, but those reports are more about Helm on k8s than abctl‑local. [1.6.0 dataplane issue; [1.6.2 dataplane secret issues] I don’t have enough documented evidence to recommend a reinstall or version change as the first step, but if: • Pods in 
airbyte-abctl
are clearly unhealthy, and • The token‑alignment workaround doesn’t help, then a clean 
abctl local uninstall
/ 
abctl local install
is a known way others have recovered from broken abctl deployments. [Unending helm debug / reinstall] If you can paste (sanitized): • The output of 
kubectl -n airbyte-abctl get pods
from inside 
airbyte-abctl-control-plane
• Any 
Unauthorized
/ 401 lines from the worker / workload‑api‑server logs we can line that up more precisely with the known 401/workload issues in the references above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai this is what I got running abctl local status:
root@airbyte-test-server:~/airbyte# abctl local status INFO Using Kubernetes provider: Provider: kind Kubeconfig: /root/.airbyte/abctl/abctl.kubeconfig Context: kind-airbyte-abctl SUCCESS Found Docker installation: version 27.5.1 SUCCESS Existing cluster 'airbyte-abctl' found ERROR Failed to initialize 'local' command ERROR unable to initialize local command: error communicating with kubernetes: unable to fetch kubernetes server version: Get "https://127.0.0.1:37799/version": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-05-18T021309Z is after 2026-05-14T170428Z INFO An error occurred while communicating with the Kubernetes cluster. If this error persists, you may need to run the "abctl local uninstall" command before attempting to run the "abctl local install" command again. Your data will persist between the uninstall and install commands. INFO A new release of abctl is available: v0.24.0 -> v0.30.4 Updating to the latest version is highly recommended
Looks like the certificate expired 5/14 at 5pm? Which makes sense because that was the day that we had our issues. How can we fix this, and why does this "certificate" matter at all? We would like to treat airbyte as a black box and this breaks the boundary. Be concise in your response, and only provide either thorough solution steps, or other commands to execute to diagnose the issue further.
k
You’ve correctly identified the root cause: the kind / Kubernetes API server certificate inside the abctl-managed cluster has expired, so 
abctl
can no longer talk to the cluster: > 
tls: failed to verify certificate: x509: certificate has expired or is not yet valid
The knowledge sources don’t document a way to rotate/renew that cert in-place. The only documented remediation for a broken abctl cluster is to *tear down and recreate the local cluster*; abctl is explicitly designed to own that kind cluster and not expose its internals as a supported surface. [abctl FAQ] Given your constraints (“treat Airbyte as a black box”) and the docs we have, the only thorough, supported fix I can point to is: Fix: Recreate the abctl cluster (data preserved) abctl’s own guidance when it cannot communicate with the cluster is exactly what you’re seeing in the status output: > “If this error persists, you may need to run the 
abctl local uninstall
command before attempting to run the 
abctl local install
command again. > Your data will persist between the uninstall and install commands.” [abctl local install] Steps: 1. *Uninstall the current local instance (keeps data):*abctl local uninstallThis stops the kind cluster and Helm release but does not delete the data directory (
~/.airbyte/abctl/data
) when run without 
--persisted
. [Uninstall Airbyte] 2. *Reinstall Airbyte locally:*abctl local installOptionally add your previous flags (e.g. 
--port
, 
--chart-version
) if you used them before. This will: ◦ Create a fresh kind cluster with a new Kubernetes API cert ◦ Reinstall the Airbyte Helm chart ◦ Reuse the existing data volumes so your workspaces, connections, etc. are preserved (per docs). [Uninstall Airbyte] 1. *Verify status:*abctl local statusYou should no longer see the TLS/certificate error, and it should report the Helm chart and app versions plus the URL (e.g. 
http://localhost:8000
). [abctl status example] 2. *(Recommended) Upgrade abctl itself:*Your output shows: “A new release of abctl is available: v0.24.0 -> v0.30.4. Updating to the latest version is highly recommended.”The docs explicitly recommend upgrading abctl via:curl -LsfS https://get.airbyte.com | bash # or, if you installed via Go or Homebrew, use the corresponding upgrade command abctl version[Upgrade abctl]Doing this before the reinstall is ideal, so the new cluster is created with the latest abctl behavior. Why this certificate matters (in abctl’s model) From the docs: • 
abctl
*always creates and manages its own kind cluster inside Docker*; it does not target an existing Kubernetes cluster. [abctl FAQ] • All 
abctl local ...
commands talk to that cluster’s Kubernetes API using the kubeconfig at 
~/.airbyte/abctl/abctl.kubeconfig
. [abctl FAQ] When the API server’s TLS cert expires, any client (including 
abctl
) that verifies TLS will refuse to connect, so: • 
abctl local status
, 
install
, 
deployments
, etc. can’t talk to Kubernetes • The guidance in the docs is to uninstall and reinstall the local instance when communication with the cluster is broken. [abctl troubleshoot – connection refused; [abctl uninstall/install] There is no documented, supported “black-box” way to rotate that cert in-place; the supported boundary is “abctl manages the cluster; if it’s broken, uninstall + install.” If you want to diagnose further (optional) If you’d like to confirm the cluster is otherwise intact before reinstalling, the only additional commands the docs suggest are: • Use 
kubectl
with the abctl kubeconfig:kubectl --kubeconfig ~/.airbyte/abctl/abctl.kubeconfig --namespace airbyte-abctl get pods[abctl FAQ – using kubectl] But given the TLS error on the API server, this is expected to fail in the same way until the cluster is recreated. Summary: Based on the docs we have, the supported fix for your expired kind/Kubernetes cert in an abctl-managed local install is: 1. Upgrade 
abctl
to the latest. 2. 
abctl local uninstall
(without 
--persisted
). 3. 
abctl local install
. 4. Re-check 
abctl local status
and your connections. This keeps your data directory but recreates the cluster and its certificates. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
3 Views
Open in Slack
PreviousNext
Powered by