This message was deleted.
# ask-ai
s
This message was deleted.
k
These logs are from the Workloads “launcher” pipeline that starts and tracks a SYNC job (replication) in Airbyte’s new architecture. [[Jobs overview](https://docs.airbyte.com/platform/understanding-airbyte/jobs)] Step‑by‑step: 1. BUILD
APPLY Stage: BUILD
– the launcher prepares the workload configuration (what to run, images, resources, etc.). 2. CLAIM
APPLY Stage: CLAIM
+
Claimed: true …
– this launcher instance successfully “claims” responsibility for running this workload so other launchers don’t run it concurrently. [[S3 dedupe issue](https://github.com/airbytehq/airbyte/issues/63359)] 3. LOAD_SHED
APPLY Stage: LOAD_SHED
– internal step where the launcher can decide to delay/start jobs based on capacity limits (
MAX_SYNC_WORKERS
, etc.). [[Jobs overview](https://docs.airbyte.com/platform/understanding-airbyte/jobs)] 4. CHECK_STATUS
APPLY Stage: CHECK_STATUS
/
No pod found running for workload …
– the launcher checks if a pod is already running for this workload. None is found, so it proceeds to create one (this “no pod found” is expected for a first attempt and not an error). You can see the same pattern in other workload logs. [[Sheets issue](https://github.com/airbytehq/airbyte/issues/56964); [S3 dedupe issue](https://github.com/airbytehq/airbyte/issues/63359)] 5. MUTEX
APPLY Stage: MUTEX
/
Mutex key … Attempting to delete existing pods… found no existing pods
– Airbyte enforces that only one pod per connection (or mutex key) is active at a time. It attempts to delete any old pods with that key; in your case there were none, so it continues. [[Local CSV issue](https://github.com/airbytehq/airbyte/issues/55726); [BigQuery/Jira issue](https://github.com/airbytehq/airbyte/issues/58082)] 6. ARCHITECTURE
APPLY Stage: ARCHITECTURE
– the launcher chooses how the pod will be structured (init container + source, destination, orchestrator/sidecar containers). [[Jobs overview](https://docs.airbyte.com/platform/understanding-airbyte/jobs)] 7. LAUNCH
APPLY Stage: LAUNCH
+ the lines with
[initContainer]
,
[source]
,
[destination]
,
[orchestrator]
– Kubernetes replication pod is created with: - init container:
workload-init-container:2.0.0
- source connector image:
source-google-ads:4.0.2
- destination connector image:
destination-s3-data-lake:0.3.41
- container orchestrator image:
container-orchestrator:2.0.0
This matches the documented model where a SYNC workload runs source, destination and orchestrator in one pod. [[Jobs overview](https://docs.airbyte.com/platform/understanding-airbyte/jobs)] 8. Workload state update
Attempting to update workload … to LAUNCHED.
/
Pipeline completed for workload …
– the launcher has successfully created the pod and marks the workload status as
LAUNCHED
in the Workload API, ending the launcher pipeline for this attempt. [[S3 dedupe issue](https://github.com/airbytehq/airbyte/issues/63359)] 9. Post replication operations
----- START POST REPLICATION OPERATIONS -----
/
No post-replication operation(s) to perform.
– after the sync finishes, Airbyte checks if there are any configured post‑replication actions (e.g., transformations, custom hooks). In your case, nothing is configured so it logs that and ends. (The sources do not describe this block explicitly, but it is consistent with the job lifecycle.) So overall: these logs show a normal, successful launch of a replication workload (Google Ads → S3 Data Lake). Nothing in the snippet indicates an error; it documents the launcher preparing, claiming, ensuring no conflicting pods, launching the pod, and then finishing, with no extra post‑replication steps to run.
a
@kapa.ai there was no replication pod created and the attempts table shows the following error { "sync": { "failures": [ { "metadata": {}, "timestamp": 1764029965532, "failureType": "system_error", "failureOrigin": "airbyte_platform", "externalMessage": "Workload failed, source: workload-monitor-heartbeat", "internalMessage": "Airbyte could not track the sync progress. Sync process exited without reporting status." } ], "standardSyncSummary": { "streamCount": 0, "streamStats": [] }, "webhookOperationSummary": { "failures": [], "successes": [] } } }
k
A message from kapa.ai
The workload launcher intended to create a replication pod (you see the LAUNCH stage and the pod spec being logged), but the sync never actually progressed to a running pod and the workload-monitor timed out / lost track of it. From the data you shared:
Copy code
json {   "failureType": "system_error",   "failureOrigin": "airbyte_platform",   "externalMessage": "Workload failed, source: workload-monitor-heartbeat",   "internalMessage": "Airbyte could not track the sync progress. Sync process exited without reporting status." }
This matches known “workload-monitor-heartbeat” failures where: - Airbyte launches (or tries to launch) the workload, - But the underlying process/pod exits, never starts correctly, or can’t be observed, - So the workload monitor cannot track heartbeat/progress and fails the sync with
Airbyte could not track the sync progress. Sync process exited without reporting status.
[[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812); [amazon-seller-partner cloud issue](https://github.com/airbytehq/airbyte/issues/53167)] In similar Kubernetes deployments, this has been seen when: - the pod creation fails validation (e.g. invalid volumes / missing secrets) so the pod never actually appears, while the platform side just sees “no pod / no heartbeat” and fails. [[Launcher orchestrator failed](https://discuss.airbyte.io/t/3823)] - or the workload-launcher / storage connection is degraded and jobs cannot be processed correctly, leading to syncs that “never start” and UI warnings:
Airbyte could not start the sync process or track the progress of the sync.
[[workload-launcher liveness issue](https://github.com/airbytehq/airbyte/issues/57011); [all sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] What your logs + attempt error mean together: 1. Launcher pipeline ran through BUILD → CLAIM → MUTEX → ARCHITECTURE → LAUNCH, and logged the intent to create
replication-job-20012-attempt-0
. 2. No replication pod ever became visible/running in the cluster (as you observed). 3. Workload monitor timed out / could not observe progress, so the attempt was marked as: -
failureType: system_error
-
failureOrigin: airbyte_platform
- with the “could not track the sync progress / exited without reporting status” message, and no streams processed (
streamCount: 0
). [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812); [amazon-seller-partner cloud issue](https://github.com/airbytehq/airbyte/issues/53167)] From the provided sources, concrete checks you can do next (all are consistent with similar issues, but exact root cause isn’t in the docs): - Look in Kubernetes events and replica set logs around
replication-job-20012-attempt-0
to see if pod creation failed validation (like the “airbyte-secret” / volume errors in the forum example). [[Launcher orchestrator failed](https://discuss.airbyte.io/t/3823)] - Confirm the workload-launcher pod is healthy and not a “zombie” after any storage / network disruptions, since that can cause jobs to enqueue but never actually start. [[workload-launcher liveness issue](https://github.com/airbytehq/airbyte/issues/57011)] - Check for any platform-wide stuck state where workloads are claimed but not actually getting pods (similar to the “no new job pods are launched / claimed: false” scenario). [[airbyte services stuck](https://github.com/airbytehq/airbyte/issues/68115)] Because the knowledge sources don’t include your exact log lines or cluster state, I can’t say precisely why the pod wasn’t created — only that, from Airbyte’s perspective, this error means “the platform launched a workload, but never saw a running sync process to track, so it failed the attempt as a system/platform error.”
a
@kapa.ai the replication pod was not created at all and the workload launcher and worker are running without any disruption.
k
A message from kapa.ai
Given what you’ve added (“replication pod was not created at all and the workload launcher and worker are running without any disruption”), the key point is: - The launcher pipeline completed logically (you see
APPLY Stage: LAUNCH
and the images listed) - But Kubernetes never ended up with a
replication-job-...
pod
, and - The attempt then failed with a platform/system error:
Workload failed, source: workload-monitor-heartbeat. Airbyte could not track the sync progress. Sync process exited without reporting status.
From the available sources, this pattern matches situations where from the Airbyte platform’s perspective, the workload was launched, but no actual runnable pod/process ever materialized for the monitor to track, so the monitor fails the attempt with that generic system error. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812); [workload-launcher liveness](https://github.com/airbytehq/airbyte/issues/57011)] However, the knowledge sources do not contain a case that exactly matches: - launcher + worker healthy, - log shows LAUNCH and images, - but no pod object at all is created in Kubernetes, - and there is no logged validation error like missing volumes/secrets. (The closest example we have is where pod creation is rejected by the API with a 422 error due to invalid spec. [[Launcher orchestrator failed](https://discuss.airbyte.io/t/3823)]) So, based solely on the docs/issues provided: - Airbyte is treating this as a platform/system failure (hence
failureOrigin: airbyte_platform
,
system_error
), because it never observed a running sync process or progress. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] - The root cause of “no pod created at all” with a healthy launcher/worker is not explicitly documented in the sources you provided, so I can’t say definitively why the pod is missing. Given that limitation, the only concrete, source‑backed guidance I can offer is: 1. Check for Kubernetes‑side pod creation errors In a similar case, the K8s API rejected pod creation (422 “Pod is invalid” due to missing secret/volume), which showed only in the launcher logs as a
Failed to create pod ...
with a detailed
KubernetesClientException
. [[Launcher orchestrator failed](https://discuss.airbyte.io/t/3823)] - If you don’t see such an error in
workload-launcher
logs for your workload ID, the sources do not give a clear next documented cause. 2. Correlate with the “all sync jobs hanging / workload-monitor-heartbeat” pattern The error text in your attempt JSON is the same generic error used when syncs get stuck in `Queued`/`Starting` and never actually run, even though the UI and launcher think they should. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] That issue confirms: this is a known failure mode where jobs do not get a proper running process, but it doesn’t document an exhaustive list of underlying Kubernetes causes. 3. **Note what we can’t confirm from the sources** - There’s no explicit documentation tying this pattern to, for example, certain Helm values, RBAC issues, or a specific launcher bug in 2.0. - There’s no mention of a bug where the launcher logs a LAUNCH step but silently never calls the Kubernetes API. So the most accurate statement I can make is: > The logs and attempt error show that Airbyte’s workload pipeline logically launched the sync, but the workload monitor never saw a running pod/process, so it failed the attempt with a platform/system error. The provided knowledge sources do not explicitly explain a scenario where this happens while the launcher and worker are healthy and no replication pod object is ever created in Kubernetes, so the exact root cause cannot be determined from these sources alone.