<@U04S8PLJWMT> When a sync fails (e.g., due to an ...
# replication-ask-ai
j
@kapa.ai When a sync fails (e.g., due to an image pull failure from the registry) with zero records synced, Airbyte is still reporting the sync status as Successful. It should be marked as Failed so failures aren't missed. Why would that be the case ?
k
Thinking completed This behavior is a known platform orchestration bug in Airbyte where early pre-launch failures (such as pod launch failures, image issues, or container initialization errors) fail to propagate correctly to the job status, causing the UI and API to report the sync as Successful (or
synced
) with 0 records loaded [Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. Why this happens While the underlying workload launcher recognizes the failure and marks the workload internally as
FAILED
, the status translation back into the sync workflow fails to handle it properly [Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]: 1. Terminal state polling: In the command workflow (
ApiCommandBase.isTerminal
), the polling loop only checks for terminal statuses like
COMPLETED
or
CANCELLED
. Because there is no explicit
FAILED
branch, the loop exits without raising an error [Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. 2. Missing attempt summary fallback: When a pod fails before running (e.g., during initialization, image pulling, or creation), the workload API returns no attempt summary (
attemptSummary == null
) and no command failures. Airbyte's replication command deserializer defaults to an empty summary with
status = null
and falls through to produce a success-shaped output without a
FailureReason
[Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. 3. Dispatch fallback to success: In
ConnectionManagerWorkflowImpl
, the dispatcher checks if
summary != null && status == FAILED
to mark a job as failed. Because the status is
null
, it falls through the conditional checks to
reportSuccess
[Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. 4. Final status reporting: The connection handler reads the resulting
lastSyncJobStatus = SUCCEEDED
and exposes the status as
synced
, even though 0 records were transferred and the container never ran [Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. Next steps & workarounds • Check Workload Launcher / Kubernetes Logs: You can inspect the root cause (such as image pull errors or node resource constraints) directly via
kubectl
[Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]:kubectl get events --namespace airbyte-namespace --field-selector reason=FailedCreate kubectl logs -l app.kubernetes.io/name=workload-launcher --namespace airbyte-namespace --tail=100 • Follow the Tracking Issue: This platform defect is being actively tracked under GitHub Issue #72448 [Replication pod creation fails (KubeClientException) but UI/Public API still reports sync as successful]. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
j
@kapa.ai what should we do to fix this
k
Based on the root-cause investigation shared in the GitHub issue thread, there is no released fix yet, but a contributor has proposed a concrete, three-part defense-in-depth fix for the platform code. Since this is a platform-level bug (not something you can fix on your end), the path forward is to track and support this proposed fix upstream [Replication pod creation fails but UI reports success]: Proposed fix (three sites) 1.
ApiCommandBase.isTerminal
— Add a
FAILED
branch so the command polling loop doesn't treat a failed workload as terminal-success (currently it only checks for
COMPLETED || CANCELLED
) [Replication pod creation fails but UI reports success]. 2.
ReplicationCommand.getOutput
— Emit a synthetic
FailureReason
when
commandOutput.attemptSummary == null && commandOutput.failures == null
, instead of silently falling through to a success-shaped
ConnectorJobOutput
[Replication pod creation fails but UI reports success]. 3.
ConnectionManagerWorkflowImpl.getFailStatus
/
getCancelledStatus
— Fall back to checking
standardSyncOutput.failures.isNotEmpty()
as a tiebreaker when
summary.status
is
null
, rather than only checking
summary != null && status == FAILED
[Replication pod creation fails but UI reports success]. The contributor noted the narrowest single change that would stop the "silent success" is fix #3 (the
getFailStatus
tiebreaker), while the full three-part fix provides more robust coverage across all the failure paths [Replication pod creation fails but UI reports success]. What you can do now • Track the issue: Follow GitHub Issue #72448 for updates on when/if this fix lands, since it is still open and unresolved as of the latest comments. • Set up a monitoring workaround: Until the platform fix is released, you can detect these false-success syncs by alerting on jobs where
records_committed = 0
despite the connection running, or where the
output
column in the
attempts
table contains a
failures
array even though the job status is
succeeded
— this pattern was specifically identified in the thread as a discriminator [Replication pod creation fails but UI reports success]. • If you're on Airbyte Cloud or have a support contract, consider opening a support ticket referencing issue #72448 to escalate visibility on your specific case. I'm not aware of any released patch version that already contains this fix based on the knowledge sources — the fix was still in the "draft PR" planning stage as of the latest activity on the issue. šŸ‘ 0 button šŸ‘Ž 0 button ā„¹ļø Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.