This message was deleted.
# ask-ai
s
This message was deleted.
k
There isn’t a single “remediation runbook” for failed syncs documented in the provided sources, but they do show how failures are typically handled and which levers you can use. At a high level, to remediate a failing connection between a custom API source (Connector Builder) and a Postgres destination on OSS 0.63.2: 1. Inspect the job logs to find failure origin - Check the latest job for the connection and look for: -
Failure Origin: source
vs
Failure Origin: destination
. - Any
ConfigErrorException
/ schema validation errors / DB errors. - Example: a Postgres destination failure showed a config error on
password
type and was fixed by correcting connector settings in the UI and re‑checking the connector.[[Postgres to Postgres failing](https://discuss.airbyte.io/t/2348)] 2. If the destination (Postgres) is failing Common causes in the sources: - Bad or drifted destination config After upgrades or helm changes, the stored JSON config can become incompatible (wrong types, missing files). The fix is to open the destination in the UI, re‑enter credentials, and run “Check connection”; recreate the destination if needed.[[Postgres to Postgres failing](https://discuss.airbyte.io/t/2348); [MySQL over SSH -> Postgres](https://discuss.airbyte.io/t/1470)] - Temporary tables and disk usage When large syncs to Postgres fail, data may sit in
_airbyte_tmp_*
tables and consume storage until a successful job cleans them up.[[Source S3 space usage](https://discuss.airbyte.io/t/2067)] Remediation: - Connect to the destination DB. - List and manually drop
_airbyte_tmp_*
tables for that connection/schema. - Ensure you have enough free disk before re‑running. - Version/platform incompatibility Using a destination image that requires a newer platform will fail with messages like:
You must upgrade your platform version to use this connector version. Either downgrade your connector or upgrade platform to 0.63.7
.[[Dest Postgres platform version](https://github.com/airbytehq/airbyte/issues/54691)] On 0.63.2 you must: - Either upgrade the Airbyte platform to a compatible version (≥0.63.7 in that example), or - Use an older destination-postgres version that is supported by 0.63.2. 3. If the source (API via Connector Builder) is failing - For other sources, failed checks or syncs show messages like “Checking source connection failed – please review this connection’s configuration”.[[GA -> Postgres failed mode](https://discuss.airbyte.io/t/3628)] - For your custom API connector, follow the same pattern: - Open the source in the UI, verify all auth/URL/params, and run Check connection. - If config has changed in your API (auth, response shape, etc.), update the connector builder config accordingly. - If you recently upgraded the connector or platform, confirm the new schema is still valid (e.g., no JSON‑schema validation errors similar to those seen in other connectors).[[MySQL to Postgres failing](https://discuss.airbyte.io/t/3258)] 4. When to reset vs just re‑run - For CDC sources (e.g. Postgres) there are specific messages about invalid offsets and recommendations to reset the connection and then ensure WAL retention and sync frequency are sufficient.[[CDC invalid position](https://github.com/airbytehq/airbyte/issues/49802)] - For a custom API source (non‑CDC), resetting usually means: - From the connection page, click Reset connection to clear state. - Run a full resync to rebuild the destination data, then let regular incremental syncs resume. 5. General remediation loop For each failing connection: 1. Open the failing job and identify: - Failure origin (source/destination). - First clear error line (schema validation, config error, DB/network error). 2. Fix configuration at the failing side: - Re‑save connector settings, or recreate source/destination if config files may be missing or corrupted.[[MySQL over SSH -> Postgres](https://discuss.airbyte.io/t/1470)] 3. Clean up destination side effects if needed: - Drop
_airbyte_tmp_*
tables for large failed jobs.[[Source S3 space usage](https://discuss.airbyte.io/t/2067)] 4. Decide whether to: - Re‑run (for transient/network issues), or - Reset connection then run a fresh sync (for state/offset or schema drift issues).[[CDC invalid position](https://github.com/airbytehq/airbyte/issues/49802)] 5. Monitor logs for the next run to confirm it completes and tmp tables are cleaned automatically. The knowledge sources don’t include connector‑builder‑specific remediation steps beyond this general pattern; they also don’t document any special remediation workflow introduced in 0.63.2. If you can share the exact error from your failing job log (first “ERROR” with stack trace and failure origin), it would be possible to map it more precisely to one of the documented patterns above.
A message from kapa.ai
For OSS 0.63.2 on‑prem (non‑Enterprise), there is no built‑in email alerting, but there is some monitoring and webhook‑based notifications: 1. Monitoring & logs (all OSS editions) - Airbyte provides detailed logs per connection that you can use for monitoring via your own tooling.[[Monitoring Airbyte](https://docs.airbyte.com/platform/operator-guides/collecting-metrics)] - Airbyte Cloud uses Datadog internally; for self‑managed you can integrate Datadog only via OpenTelemetry (see below).[[Monitoring Airbyte](https://docs.airbyte.com/platform/operator-guides/collecting-metrics)] 2. Webhook notifications for failed syncs (Self‑Managed / OSS) - Self‑managed deployments (including OSS) can send notifications to a webhook for events such as failed syncs, successful syncs, repeated failures, etc.[[Manage notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications)] - Email notifications are Cloud‑only; self‑managed cannot send emails directly.[[Manage notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications)] 3. OpenTelemetry metrics (Self‑Managed Enterprise only) - If you run Self‑Managed Enterprise (including on OpenShift), you can enable OpenTelemetry metrics export (no traces/logs) to an OTEL collector, then drive alerting from your monitoring stack (e.g. Datadog, Prometheus, etc.).[[OpenTelemetry metrics](https://docs.airbyte.com/platform/operator-guides/open-telemetry); [Monitoring Airbyte](https://docs.airbyte.com/platform/operator-guides/collecting-metrics)] - This feature is explicitly Enterprise‑only, not available in Core/Standard/OSS editions.[[OpenTelemetry metrics](https://docs.airbyte.com/platform/operator-guides/open-telemetry)] So: - OSS 0.63.2: - Has logs and connection status. - Can send webhook notifications for failed syncs if you configure workspace notifications. - No built‑in email alerts and no built‑in metrics export unless you are on Enterprise. - Enterprise on OpenShift: - Same webhook notifications. - Plus OpenTelemetry metrics export, from which you can build monitoring and alerting in your observability stack.
A message from kapa.ai
For Airbyte OSS / self‑managed (including 0.63.2), you do get monitoring/alerting via webhook notifications (no built‑in email). ### What OSS supports The “Cloud vs Self‑Managed” docs explicitly say: - Airbyte Cloud: email + webhook - **Self‑Managed: webhook only**[[Manage notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications)] Events you can be notified about via webhook include (for OSS): - Failed syncs - Successful syncs - Connection Updates - Connection Updates Requiring Action - Warning – Repeated Failures - Sync Disabled – Repeated Failures[[Manage notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications)] So OSS does support alerting for failed syncs through webhooks. ### How webhooks work in OSS Webhooks are configured per workspace. Conceptually: 1. In the workspace notification settings, you provide a Webhook URL (any HTTP endpoint you control). 2. You toggle which events should send notifications to that webhook (e.g. Failed syncs).[[Manage notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications)] When one of those events occurs, Airbyte sends an HTTP POST with a JSON body to your webhook. The docs give example payloads (same for Cloud and Self‑Managed): #### Failed sync payload
Copy code
json {   "data": {     "workspace": {       "id": "b510e39b-e9e2-4833-9a3a-963e51d35fb4",       "name": "Workspace1",       "url": "https://link/to/ws"     },     "connection": {       "id": "64d901a1-2520-4d91-93c8-9df438668ff0",       "name": "Connection",       "url": "https://link/to/connection"     },     "source": {       "id": "c0655b08-1511-4e72-b7da-24c5d54de532",       "name": "Source",       "url": "https://link/to/source"     },     "destination": {       "id": "5621c38f-8048-4abb-85ca-b34ff8d9a298",       "name": "Destination",       "url": "https://link/to/destination"     },     "jobId": 9988,     "startedAt": "2024-01-01T00:00:00Z",     "finishedAt": "2024-01-01T01:00:00Z",     "bytesEmitted": 1000,     "bytesCommitted": 90,     "recordsEmitted": 89,     "recordsCommitted": 45,     "errorMessage": "Something failed",     "errorType": "config_error",     "errorOrigin": "source",     "bytesEmittedFormatted": "1000 B",     "bytesCommittedFormatted": "90 B",     "success": false,     "durationInSeconds": 3600,     "durationFormatted": "1 hours 0 min"   } }
[[Set up webhook notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] Key fields you can use for alerting/automation: -
success
(false when failed) -
errorMessage
,
errorType
,
errorOrigin
-
workspace
,
connection
,
jobId
for routing to teams or dashboards. #### Successful sync payload Same structure, but without error fields and with
success: true
and fully committed bytes/records.[[Set up webhook notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] #### Other events For some events (e.g., Connection Updates, Repeated Failures, Upgrade Required), the docs note: > “Webhook doesn’t contain payload and only works for Slack notifications.”[[Set up webhook notifications](https://docs.airbyte.com/platform/cloud/managing-airbyte-cloud/manage-airbyte-cloud-notifications#set-up-webhook-notifications)] So for those, your endpoint may just receive Slack‑formatted text blocks (and no
data
object). ### Using webhooks for monitoring/alerting in OSS In OSS, you typically: - Point the Webhook URL at: - A Slack Incoming Webhook (if you just want Slack alerts), or - Your own small service (or a generic gateway) that: - Parses the JSON payload, - Filters on
success: false
and specific connections, - Forwards to email/SMS/incident tools (PagerDuty, Opsgenie, etc.). This is the only built‑in alerting mechanism documented for OSS; there’s no native email for self‑managed instances, and the docs don’t show any additional OSS‑only alerting features beyond webhooks.