https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Airbyte v1 sync failing with error ...
# replication-ask-ai
a
@kapa.ai Airbyte v1 sync failing with error
Copy code
Warning from source: Workload failed, source: workload-monitor-start

Airbyte could not start the process within time limit. The workload was claimed but never started.
k
The knowledge sources show this exact error happening in Airbyte 1.x, but they do not provide a definitive root cause or a standard fix, so the guidance below is necessarily limited. The error: > 
Airbyte could not start the process within time limit. The workload was claimed but never started.
means: • The workload-launcher successfully claimed the job from the queue. • But the corresponding Kubernetes pod / process never actually started within the timeout window, so the workload-monitor marked it as failed. [MySQL source issue] Similar symptoms appear in other issues where: • Jobs stay in 
Queued
/ 
Starting
and never progress. [Jobs hanging] • The launcher or monitor is alive but unable to correctly process work (e.g., storage/network issues, “zombie” launcher). [Launcher liveness / storage] • The system reports “workload was never claimed” or “never started” and no pods are created. [Replication pod creation fails] From these, the most likely categories of problems are: 1. Kubernetes / infrastructure issues ◦ Replication pods cannot be created (cluster quota, node pressure, PodSecurityPolicy, etc.). ◦ Network/storage problems between 
workload-launcher
and your backing storage (S3/GCS/Azure/etc.) causing the launcher to get stuck while still passing liveness probes. [Launcher liveness / storage] 2. Workload system stuck ◦ Workloads are claimed but never transition to a running pod, so the monitor times them out. [MySQL source issue] Because the sources don’t include a concrete, step‑by‑step fix, I can’t give you a guaranteed resolution, but you can systematically narrow it down: 1. Check for pod creation failures ◦ Look in your cluster for recent pods in the Airbyte namespace: ▪︎ Are new 
replication-*
, 
check-*
, or 
discover-*
pods being created at all? ▪︎ If they appear and immediately go to 
Error
/ 
CrashLoopBackOff
/ 
Pending
, inspect their events and logs. ◦ If no pods are created for new syncs, you may be hitting the same class of issue as “replication pod creation fails but UI says success”. [Replication pod creation fails] 2. Inspect 
workload-launcher
and 
workload-monitor
logs ◦ Look for: ▪︎ Errors talking to your object storage (S3/GCS/Azure, etc.). ▪︎ Kube client exceptions when trying to create pods. ▪︎ Repeated “Checking for not started / not claimed workloads” without corresponding pod starts. [Temporal context exceeded] 3. Verify storage connectivity ◦ If you use external storage (e.g., Azure storage in the example), confirm: ▪︎ Network connectivity from the 
workload-launcher
pod to the storage endpoint. ▪︎ Credentials and configuration for 
global.storage.type
and related settings. [Launcher liveness / storage] 4. Check cluster capacity / limits ◦ Ensure: ▪︎ Sufficient CPU/memory and no exhausted pod quotas. ▪︎ No restrictive PodSecurity / NetworkPolicy changes that would block job pods. Because the knowledge sources don’t document a specific configuration bug or patch for this exact message in v1, if you still see the error after checking the above, I’d recommend: • Collecting: ◦ Logs from 
workload-launcher
, 
workload-monitor
, and the Airbyte 
worker
service around the failing sync. ◦ 
kubectl describe pod
/ events for any related job pods. • Opening or updating a GitHub issue with those details, referencing that you’re seeing “claimed but never started” behavior similar to the MySQL and hanging-jobs reports. [MySQL source issue; Jobs hanging] If you share: • Your deployment type (Helm chart version, Airbyte version, cloud/K8s provider), and • Whether you see replication/check pods being created at all, I can help you map that more directly to the closest known issue pattern in these reports. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai deployed using abctl on a single VM
k
The knowledge sources don’t describe this exact error specifically for abctl single‑VM installs, so some of the following is inferred from general abctl troubleshooting guidance; I’ll call that out as uncertainty where relevant. You’re seeing, on Airbyte v1: > 
Workload failed, source: workload-monitor-start
> 
Airbyte could not start the process within time limit. The workload was claimed but never started.
On v1, this means the workload system claimed the job, but the underlying process/pod never actually started before the timeout. A very similar symptom is reported for a MySQL source in v1: the sync hangs for hours and then fails with the same message. [MySQL source issue] Because you’re on abctl / single VM, the relevant official docs are the generic abctl troubleshooting pages, which focus on deployment and cluster‑level issues (bootloader, kind cluster, timeouts, etc.), not this specific runtime workload error. [Troubleshoot abctl] Given that limitation, here’s what you can concretely do on an abctl single‑VM install: 1. Check if job pods are being created at all Even on a single‑VM abctl deployment, Airbyte runs on a local Kubernetes (kind) cluster. The “claimed but never started” error is consistent with no job pod ever starting. On the VM, use 
kubectl
against the abctl cluster and check: • Are there any new 
check-*
, 
sync-*
, or 
discover-*
pods when you trigger a sync? • Do they stay in 
Pending
or 
ContainerCreating
, or do they never appear? If you see no pods at all for the job, that matches the pattern where the workload is claimed but pod creation fails or is blocked (similar to the “replication pod creation fails” issue, though that one is on a remote K8s cluster). [Replication pod fails] 2. Inspect workload‑launcher / worker logs The v1 MySQL issue with the same error doesn’t include logs, but it confirms that the failure is detected by the workload monitor when the process never starts. [MySQL source issue] On your VM, grab logs from: • 
workload-launcher
• 
worker
(and, if present, 
workload-monitor
) Look around the time of the failing sync for: • Kube client errors when creating pods • Timeouts or resource errors • Any mention of “workload was claimed but never started” or similar The abctl troubleshooting docs don’t give specific commands for this runtime case, but they do emphasize using logs to diagnose failures in the abctl‑managed cluster. [Troubleshoot abctl] 3. Check VM / kind cluster capacity The docs show that abctl spins up a kind cluster inside Docker on the VM and that installation can fail with generic “timed out waiting for the condition” when the cluster can’t progress. [Troubleshoot abctl] While they don’t explicitly tie this to workload‑monitor errors, on a single VM it’s plausible that: • The VM is low on CPU/RAM/disk, so new job pods can’t be scheduled or started. • Docker/kind is constrained, causing pods to remain pending or never start. So on the VM, check: • Overall CPU/RAM usage • Disk space (especially where Docker stores images/volumes) • 
kubectl get pods -A
to see if any system pods are in 
Pending
/ 
Error
4. If the problem persists Because the knowledge sources don’t document a known bug or configuration fix for this specific combination (v1 + abctl + “claimed but never started”), I can’t point you to a definitive patch. If, after the checks above, you still see: • Workloads being “claimed” in the UI/logs • But no corresponding job pods starting in the abctl cluster then the next step would be to: 1. Collect: ◦ 
kubectl get pods -A
and 
kubectl describe
for any job pods ◦ Logs from 
workload-launcher
and 
worker
around the failing run 2. Open a GitHub issue referencing that you’re on abctl single‑VM and seeing the same error as in the MySQL issue (“workload was claimed but never started”). [MySQL source issue; Troubleshoot abctl] If you can share: • Your exact Airbyte version (e.g., 1.8.x) • abctl version • Output of 
kubectl get pods -A
when a sync is running I can help you interpret those against the patterns in the existing issues. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai OK I've airbyte db as external database if I create a new instance and connect to that db will connections pick up from the last state in older instance
k
Yes, if you deploy a new Airbyte instance and point it to the same external Postgres database, your connections and their states will be preserved and will continue from where they left off, as long as: • The Airbyte platform versions match (old and new instances must be on the same version). [Migrate Docker to Docker; Migrate between instances] • The new deployment is correctly configured to use that same external DB (host, DB name, user, password). [External DB integration] Relevant points from the docs and forum: • For users with an external database, a new installation can be configured to use the same database. [Migrating from Docker Compose] • Community guidance confirms that moving from one deployment type to another (e.g., Docker → K8s) is possible by reusing the same external Postgres DB. [Docker to K8S migration] • When migrating between instances, support recommends exporting/importing config or pointing the new instance to the same DB, with the assumption that jobs and states are preserved when the DB is reused. [Advice migrating instances] What this means for you: • If your current instance is stopped and the new instance uses the same external Airbyte DB, all: ◦ sources, destinations, connections ◦ job history ◦ incremental sync state will be read from that DB and should continue from the last stored state. Limitations / caveats (from the docs): • This is not supported for Docker Compose’s internal DB; the “reuse DB” guidance is explicitly for external databases. [Migrating from Docker Compose] • You must ensure *version compatibility*; community tutorials stress using the same Airbyte version when restoring/migrating via the DB. [Migrate Docker to Docker] If you tell me your current Airbyte version and how you plan to deploy the new instance (abctl, Helm, etc.), I can point you to the exact DB config section you’ll need. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai getting this message using abctl local status
Copy code
abctl local status

  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /home/adnan/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
  ERROR   Unable to create Docker client                                                                                                 
  ERROR   Unable to determine if Docker is installed                                                                                     
  ERROR   unable to determine docker installation status: error communicating with docker: unable to create client: error communicating with docker: unable to create docker client
                                                                                                                                         
  INFO    An error occurred while communicating with the Docker daemon.                                                                  
          Ensure that Docker is running and is accessible.  You may need to upgrade to a newer version of Docker.
          For additional help please visit <https://docs.docker.com/get-docker/>
  INFO    A new release of abctl is available: v0.24.0 -> v0.30.3                                                                        
          Updating to the latest version is highly recommended
k
abctl
relies on Docker being available and reachable on your machine. The error you’re seeing is exactly what 
abctl
reports when it cannot talk to the Docker daemon: > 
Unable to create Docker client
> 
Unable to determine if Docker is installed
> 
error communicating with docker
From the docs, 
abctl
is designed to run Airbyte in a kind Kubernetes cluster inside Docker, and it requires Docker (or Docker Desktop) to be installed and running before you use it. [abctl overview] The official guidance is: > “Before you use abctl, install Docker Desktop on your machine…” [abctl overview] and the troubleshooting page reiterates that Docker must be working, suggesting to verify you can run containers like 
docker run hello-world
when there are Docker‑related errors. [Troubleshoot abctl] So, to resolve your 
abctl local status
error: 1. Ensure Docker is installed ◦ On Linux, you can use Docker Engine; on Mac/Windows, Docker Desktop is recommended. [abctl overview] 2. Ensure the Docker daemon is running and accessible ◦ Start the Docker service / Docker Desktop. ◦ Confirm it works by running a simple command like:
Copy code
docker version
       docker run hello-world
◦ If these fail, fix Docker first; 
abctl
will not work until Docker is healthy. [Troubleshoot abctl] 3. (Optional but recommended) Upgrade 
abctl
 ◦ Your output shows a newer version is available (
v0.24.0 -> v0.30.3
). ◦ The docs recommend staying current, especially since newer versions include fixes and better diagnostics. [Troubleshoot abctl] Once Docker is installed, running, and 
docker version
works, re‑run:
Copy code
abctl local status
If you still get Docker communication errors after confirming Docker works, the knowledge sources don’t provide more specific remediation steps beyond verifying Docker installation and version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai running with sudo I see docker is fine and a different error
Copy code
sudo abctl local status
  INFO    Using Kubernetes provider:
            Provider: kind
            Kubeconfig: /root/.airbyte/abctl/abctl.kubeconfig
            Context: kind-airbyte-abctl
 SUCCESS  Found Docker installation: version 28.0.0                                                                                      
 SUCCESS  Existing cluster 'airbyte-abctl' found                                                                                         
  ERROR   Failed to initialize 'local' command                                                                                           
  ERROR   unable to initialize local command: error communicating with kubernetes: unable to fetch kubernetes server version: Get "<https://127.0.0.1:35117/version>": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2026-02-20T18:59:39Z is after 2026-02-20T12:56:26Z
                                                                                                                                         
  INFO    An error occurred while communicating with the Kubernetes cluster.                                                             
          If this error persists, you may need to run the "abctl local uninstall" command before attempting to run the
          "abctl local install" command again.
          Your data will persist between the uninstall and install commands.
  INFO    A new release of abctl is available: v0.24.0 -> v0.30.3                                                                        
          Updating to the latest version is highly recommended
k
The error is coming from the kind Kubernetes cluster that abctl created: > 
tls: failed to verify certificate: x509: certificate has expired or is not yet valid
abctl’s own guidance for generic Kubernetes‑communication failures is exactly what you see echoed in the message: > “An error occurred while communicating with the Kubernetes cluster… you may need to run the 
abctl local uninstall
command before attempting to run the 
abctl local install
command again. Your data will persist between the uninstall and install commands.” [Install/manage abctl] Given the docs, the supported recovery path for a broken abctl‑managed cluster (including when 
local status
cannot talk to Kubernetes) is: 1. Uninstall the local Airbyte instance but keep data:
Copy code
abctl local uninstall
This “stops running all containers, but keeps your data.” [Install/manage abctl] 2. Re‑install Airbyte:
Copy code
abctl local install
This recreates the kind cluster and Helm release while reusing the persisted data. [Install/manage abctl] 3. Only if you want to wipe everything, use:
Copy code
abctl local uninstall --persisted
   rm -rf ~/.airbyte/abctl
This deletes both the deployment and all data. [Install/manage abctl] The docs do not mention a way to “fix” an expired kind/Kubernetes certificate in place for abctl; the recommended approach for cluster‑level issues is exactly this uninstall → install cycle while relying on persisted data. [Troubleshoot abctl; Install/manage abctl] Also, abctl is telling you a newer version is available. Upgrading abctl itself is recommended in the operator guide and may include fixes and better handling for these situations:
Copy code
curl -LsfS https://get.airbyte.com | bash -
abctl version
[Upgrade abctl] So, based on the official guidance: 1. Upgrade 
abctl
(optional but recommended). 2. Run 
abctl local uninstall
(without 
--persisted
) to reset the broken cluster while keeping data. 3. Run 
abctl local install
again. 4. Re‑run 
abctl local status
to confirm the cluster is healthy. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
2 Views
Open in Slack
PreviousNext
Powered by