https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> My cron syncs are failing on airbyt...
# replication-ask-ai
a
@kapa.ai My cron syncs are failing on airbyte. When I download the logs I'm not getting anything other then
> ATTEMPT 1/5
2026-07-23 041032 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=1, successivePartialFailures=0, totalCompleteFailures=1, totalPartialFailures=0) Backoff before next attempt: 10 seconds
> ATTEMPT 2/5
2026-07-23 041033 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=1, successivePartialFailures=0, totalCompleteFailures=1, totalPartialFailures=0) 2026-07-23 041033 warn Backing off for: 10 seconds. 2026-07-23 044132 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=2, successivePartialFailures=0, totalCompleteFailures=2, totalPartialFailures=0) Backoff before next attempt: 30 seconds
> ATTEMPT 3/5
2026-07-23 044133 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=2, successivePartialFailures=0, totalCompleteFailures=2, totalPartialFailures=0) 2026-07-23 044133 warn Backing off for: 30 seconds. 2026-07-23 051232 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=3, successivePartialFailures=0, totalCompleteFailures=3, totalPartialFailures=0) Backoff before next attempt: 1 minute 30 seconds
> ATTEMPT 4/5
2026-07-23 051233 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=3, successivePartialFailures=0, totalCompleteFailures=3, totalPartialFailures=0) 2026-07-23 051233 warn Backing off for: 1 minute 30 seconds. 2026-07-23 054432 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=4, successivePartialFailures=0, totalCompleteFailures=4, totalPartialFailures=0) Backoff before next attempt: 4 minutes 30 seconds
> ATTEMPT 5/5
2026-07-23 054433 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=4, successivePartialFailures=0, totalCompleteFailures=4, totalPartialFailures=0) 2026-07-23 054433 warn Backing off for: 4 minutes 30 seconds. 2026-07-23 061932 info Retry State: RetryManager(completeFailureBackoffPolicy=BackoffPolicy(minInterval=PT10S, maxInterval=PT30M, base=3), partialFailureBackoffPolicy=null, successiveCompleteFailureLimit=5, successivePartialFailureLimit=1000, totalCompleteFailureLimit=10, totalPartialFailureLimit=20, successiveCompleteFailures=5, successivePartialFailures=0, totalCompleteFailures=5, totalPartialFailures=0) Backoff before next attempt: 13 minutes 30 seconds 2026-07-23 061933 error Failing job: 7427, reason: Job failed after too many retries for connection 0aa52568-5f92-4b06-b7f3-5540cfda2b86
k
Sync Failing with No Data Synced (Complete Failures) Your logs show that every attempt is failing with zero data synchronized — this is a "complete failure" pattern. Each attempt exits immediately without producing any output, triggering the retry backoff sequence (10s → 30s → 90s → 4m30s → 13m30s) until the 
successiveCompleteFailureLimit=5
is reached. [Sync Jobs] What the logs tell us The key observation is that there are no connector logs between the attempt start and the retry state message. This means the sync is failing before the connector even starts running — likely at the platform/orchestration level (e.g., failing to create an attempt, spin up a worker pod, or reach the source/destination). This is consistent with a known pattern where the 
CreateNewAttemptNumber
activity fails: [Github Issue]
Copy code
Activity with activityType='CreateNewAttemptNumber' failed: 'Activity task failed'
...retryState=RETRY_STATE_MAXIMUM_ATTEMPTS_REACHED
Likely causes to investigate 1. Worker/pod startup issues — If you're on Kubernetes (EKS, GKE, etc.), check if the job pods are being scheduled and starting correctly. Look at your 
airbyte-worker
pod logs for errors around the time of the sync attempts. 2. Resource constraints — Insufficient CPU/memory for job containers can cause silent failures before any connector output is produced. Check 
JOB_MAIN_CONTAINER_CPU_REQUEST
/ 
JOB_MAIN_CONTAINER_MEMORY_REQUEST
settings. [Configuring Airbyte] 3. Source/destination connectivity — The worker may be failing the pre-sync source/destination check. Check 
airbyte-worker
logs for 
SOURCE CHECK: Failed
messages. 4. Temporal/workflow issues — If the Temporal service is under load or has issues, activity tasks can fail before any work is done. Recommended steps 1. Check 
airbyte-worker
pod logs at the timestamps of each attempt start (e.g., 
04:10:32
, 
04:10:33
, etc.) for errors. 2. Check if job pods are being created — run 
kubectl get pods -n <airbyte-namespace>
during a sync to see if connector pods appear. 3. Verify source and destination connectivity by manually testing the connections in the Airbyte UI. 4. Check resource availability on your cluster nodes around the time of failures. 5. If on OSS, consider temporarily reducing 
SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVE
to fail faster while debugging, so you get quicker feedback. [Configuring Airbyte] The ~30-minute gap between each attempt start and the next retry state log (e.g., 
04:10:33
→ 
04:41:32
) is also unusual — the backoff should only be 10 seconds, but ~31 minutes elapsed. This strongly suggests the attempt itself is hanging or timing out at the platform level before failing, which points to a worker or infrastructure issue rather than a connector issue. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai orchestrator pods are not starting what could be the reason
k
Orchestrator Pods Not Starting Based on the knowledge sources, here are the most common reasons orchestrator pods fail to start: 1. Pre-existing Pod in NOT_STARTED State A stale pod from a previous attempt blocks creation of a new one:
Copy code
WorkerException: Failed to create pod orchestrator-repl-job-XXXXX-attempt-0,
pre-existing pod exists which didn't advance out of the NOT_STARTED state.
Fix: Manually delete the stuck pod:
Copy code
kubectl delete pod orchestrator-repl-job-<job-id>-attempt-0 -n <airbyte-namespace>
[Forum Post] 2. Missing or Invalid Kubernetes Secret The pod spec references a secret (e.g., 
airbyte-secret
) that doesn't exist:
Copy code
KubernetesClientException: spec.volumes[1].secret.secretName: Required value
spec.containers[0].volumeMounts[1].name: Not found: "airbyte-secret"
Fix: Verify the secret exists in the correct namespace:
Copy code
kubectl get secrets -n <airbyte-namespace>
[Forum Post] 3. Pod Spec Update Conflict Kubernetes rejects an update to an existing pod because immutable fields changed (e.g., container ports, resource limits):
Copy code
Forbidden: pod updates may not change fields other than
`spec.containers[*].image`, `spec.activeDeadlineSeconds` or `spec.tolerations`
Fix: Delete the conflicting pod and let Airbyte recreate it. [Forum Post] 4. Resource / Node Constraints Nodes don't have enough CPU/memory to schedule the pod, causing it to stay in 
Pending
. Check with:
Copy code
kubectl describe nodes | grep -A 5 "Allocated resources"
kubectl get events --namespace <airbyte-namespace> --field-selector reason=FailedCreate
[Github Issue] 5. Temporal or Workload Launcher Issues If the 
workload-launcher
or 
temporal
pods are unhealthy, they won't dispatch orchestrator pods. Check:
Copy code
kubectl logs -l app.kubernetes.io/name=workload-launcher -n <airbyte-namespace> --tail=100
[Github Issue] 6. Cron Service Failure (Heartbeat Monitoring) If the 
airbyte-cron
service is hitting errors (e.g., 
RESOURCE_EXHAUSTED: namespace rate limit exceeded
), it can't clean up stuck workloads, causing new pods to never be launched:
Copy code
kubectl logs -l app.kubernetes.io/name=cron -n <airbyte-namespace> --tail=100
[Github Issue] Quick Diagnostic Checklist
Copy code
# 1. Check for stuck/pending pods
kubectl get pods -n <airbyte-namespace>

# 2. Check events for scheduling failures
kubectl get events -n <airbyte-namespace> --sort-by='.lastTimestamp'

# 3. Check workload-launcher logs
kubectl logs -l app.kubernetes.io/name=workload-launcher -n <airbyte-namespace> --tail=100

# 4. Check cron logs
kubectl logs -l app.kubernetes.io/name=cron -n <airbyte-namespace> --tail=100
Could you share the output of these commands or any error messages from the 
workload-launcher
logs? That would help narrow down the exact cause. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai kubectl describe nodes | grep -A 5 "Allocated resources" Allocated resources: (Total limits may be over 100 percent, i.e., overcommitted.) Resource Requests Limits -------- -------- ------ cpu 1050m (26%) 100m (2%) memory 380Mi (2%) 390Mi (2%)
k
Node Resources Look Fine Your node has plenty of available resources — only 26% CPU and 2% memory are requested. Resource exhaustion is not the cause of your orchestrator pods not starting. This points to a different root cause. Based on the knowledge sources, the most likely culprits are: 1. Check Workload Launcher Logs (Most Important Next Step)
Copy code
kubectl logs -l app.kubernetes.io/name=workload-launcher --namespace <airbyte-namespace> --tail=200
Look for errors like: • 
KubeClientException
— pod creation failing at the Kubernetes API level • 
pre-existing pod exists which didn't advance out of the NOT_STARTED state
— stale pod blocking creation • 
ResourceConstraintException
— misconfigured resource requests being rejected 2. Check for Stuck/Pending Pods
Copy code
kubectl get pods -n <airbyte-namespace>
kubectl describe pod <orchestrator-pod-name> -n <airbyte-namespace>
If a pod is in 
Pending
state, the 
describe
output will show the exact scheduling failure reason under 
Events
. 3. Check Kubernetes Events
Copy code
kubectl get events -n <airbyte-namespace> --sort-by='.lastTimestamp' | tail -30
4. Known Bug: Orchestrator Memory Hardcoded to 1Gi There is a known open bug where the orchestrator container ignores all memory configuration and always runs with a hardcoded 
1Gi
limit, which can cause OOM crashes. [Github Issue] A workaround reported by users is implementing a Kubernetes Mutating Webhook to patch the orchestrator container memory at pod creation time. [Github Issue] 5. Known Bug: Sync Reported as Successful Despite Pod Failure Also note there is an open bug where a 
KubeClientException
during pod creation causes the sync to fail internally, but the UI/API still reports it as successful — so the actual failure may be silently swallowed. [Github Issue] Could you share the output of the 
workload-launcher
logs and 
kubectl get pods -n <airbyte-namespace>
? That will pinpoint the exact failure reason. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai Starting pod sweeper cycle in namespace [airbyte-abctl]... 2026-07-23 105301,208 [scheduled-executor-thread-5] INFO i.a.w.l.PodSweeper(sweepPods):88 - Will sweep Succeeded pods older than 2026-07-23T104301.208Z (UTC). 2026-07-23 105301,208 [scheduled-executor-thread-5] INFO i.a.w.l.PodSweeper(sweepPods):91 - Will sweep unsuccessful pods older than 2026-07-23T085301.208Z (UTC). 2026-07-23 105301,261 [scheduled-executor-thread-5] INFO i.a.w.l.PodSweeper(sweepPods):149 - Completed pod sweeper cycle. 2026-07-23 105501,208 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):56 - Starting pod sweeper cycle in namespace [airbyte-abctl]... 2026-07-23 105501,209 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):88 - Will sweep Succeeded pods older than 2026-07-23T104501.209Z (UTC). 2026-07-23 105501,210 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):91 - Will sweep unsuccessful pods older than 2026-07-23T085501.209Z (UTC). 2026-07-23 105501,266 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):149 - Completed pod sweeper cycle. 2026-07-23 105701,208 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):56 - Starting pod sweeper cycle in namespace [airbyte-abctl]... 2026-07-23 105701,208 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):88 - Will sweep Succeeded pods older than 2026-07-23T104701.208Z (UTC). 2026-07-23 105701,209 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):91 - Will sweep unsuccessful pods older than 2026-07-23T085701.208Z (UTC). 2026-07-23 105701,265 [scheduled-executor-thread-2] INFO i.a.w.l.PodSweeper(sweepPods):149 - Completed pod sweeper cycle. 2026-07-23 105901,208 [scheduled-executor-thread-1] INFO i.a.w.l.PodSweeper(sweepPods):56 - Starting pod sweeper cycle in namespace [airbyte-abctl]... 2026-07-23 105901,208 [scheduled-executor-thread-1] INFO i.a.w.l.PodSweeper(sweepPods):88 - Will sweep Succeeded pods older than 2026-07-23T104901.208Z (UTC). 2026-07-23 105901,209 [scheduled-executor-thread-1] INFO i.a.w.l.PodSweeper(sweepPods):91 - Will sweep unsuccessful pods older than 2026-07-23T085901.208Z (UTC). 2026-07-23 105901,261 [scheduled-executor-thread-1] INFO i.a.w.l.PodSweeper(sweepPods):149 - Completed pod sweeper cycle.
k
Workload Launcher Only Running Pod Sweeper — No Workloads Being Claimed This is a critical finding. Your 
workload-launcher
logs show only 
PodSweeper
cycles running — there are no 
WorkloadApiQueuePoller
or 
ApiQueueConsumer
entries. This means the launcher is not polling for or claiming any workloads, so orchestrator pods will never be created. A healthy launcher should show lines like:
Copy code
INFO  i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with <group-id> and high
INFO  i.a.w.l.p.c.WorkloadApiQueuePoller(resumePolling) - Resuming ApiQueuePoller with <group-id> and default
The absence of these lines is the root cause of your orchestrator pods not starting. Most Likely Cause: Dataplane Group Mismatch This is a well-known issue, especially after upgrades. The workload-launcher registers under one 
dataplane_group
, but workloads are being routed to a different (old) 
dataplane_group
, so the launcher never sees any work to claim. [Github Issue] Diagnostic Steps Run these queries against your Airbyte database:
Copy code
-- Check how many dataplane groups exist
SELECT * FROM public.dataplane_group;

-- Check which group workloads are being routed to
SELECT id, status, dataplane_group FROM workload WHERE status = 'pending' LIMIT 10;

-- Check which group the launcher is registered under
SELECT d.id, d.dataplane_group_id, dg.name
FROM dataplane d
JOIN dataplane_group dg ON dg.id = d.dataplane_group_id
WHERE d.tombstone = false;

-- Check workspace dataplane group assignment
SELECT id, organization_id, dataplane_group_id FROM public.workspace;
If you see two dataplane groups and the 
workspace.dataplane_group_id
points to the old one while the launcher is registered under the new one, that confirms the mismatch. [Github Issue] Fix Update the workspace to point to the correct (active) dataplane group:
Copy code
-- Replace <correct-dataplane-group-id> with the ID the launcher is registered under
UPDATE public.workspace
SET dataplane_group_id = '<correct-dataplane-group-id>';
Then restart the workload-launcher:
Copy code
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
After the restart, you should see the 
WorkloadApiQueuePoller
lines appear in the logs, confirming the launcher is now polling for workloads. [Github Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai how to find Replace correct-dataplane-group-id with the ID the launcher is registered under
k
Finding the Correct Dataplane Group ID Run this query against your Airbyte database to find the group ID the launcher is registered under:
Copy code
SELECT d.id AS dataplane_id, 
       d.dataplane_group_id AS correct_group_id, 
       dg.name AS group_name
FROM dataplane d
JOIN dataplane_group dg ON dg.id = d.dataplane_group_id
WHERE d.tombstone = false;
The value in 
correct_group_id
 column is what you need. [Github Issue] Alternatively, you can use this safer version that moves all dataplanes to the 
AUTO
group automatically (no manual UUID needed):
Copy code
-- Move all dataplanes to the AUTO group
UPDATE dataplane
SET dataplane_group_id = (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
)
WHERE dataplane_group_id != (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
);
Then update the workspace to match:
Copy code
UPDATE public.workspace
SET dataplane_group_id = (
  SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1
);
Then restart the launcher:
Copy code
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
[Github Issue] > Note: This fix may need to be reapplied after every 
abctl local install
run, as the bootloader recreates the mismatch on each startup. [Github Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
a
@kapa.ai explain previous solution breifly
k
Brief Explanation The root cause is a dataplane group mismatch — after an upgrade or reinstall, Airbyte creates a new 
dataplane_group
in the database, but the old one is still being used to route workloads. This creates a situation where: • 🔴 Workload-launcher polls for jobs under → New group • 🔴 Workloads are routed to → Old group Since they don't match, the launcher never sees any work, so no pods are ever created. [Github Issue] The Fix (3 steps) 1. Move all dataplanes to the AUTO group (the original group):
Copy code
UPDATE dataplane
SET dataplane_group_id = (SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1)
WHERE dataplane_group_id != (SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1);
2. Update the workspace to point to the same group:
Copy code
UPDATE public.workspace
SET dataplane_group_id = (SELECT id FROM dataplane_group WHERE name = 'AUTO' LIMIT 1);
3. Restart the launcher so it re-registers under the correct group:
Copy code
kubectl rollout restart deployment/airbyte-abctl-workload-launcher -n airbyte-abctl
After the restart, the launcher will start polling for workloads and orchestrator pods will begin launching. [Github Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
6 Views
Open in Slack
PreviousNext
Powered by