@kapa.ai What does a postgres -> bigquery replication job do when it tries again after being terminated in the source system? Does it "resume" from the last failure point or try to load the stream again from scratch? This is a CDC/changelog stream reading from postgres.

For a CDC (logical replication) stream from Postgres to BigQuery, the behavior on retry depends on where in the sync process the failure occurred: During the Initial Snapshot Phase When a CDC connection first runs, it takes a full snapshot of existing data. If the job is terminated during this phase, the next attempt may re-run the snapshot from scratch — and importantly, there is a known bug where Airbyte commits the wrong LSN during the initial snapshot. If the job fails before finding any incremental changes in the subsequent CDC phase, the next job may fail or trigger a re-sync depending on your Invalid CDC Position Behavior setting. [source-postgres LSN bug] During Incremental CDC Reads (After Initial Snapshot) After the initial snapshot, Airbyte saves its position in the WAL as a state (LSN offset). On retry, it attempts to resume from that saved offset. However, a full re-sync from scratch will be triggered if the saved offset is no longer valid — specifically, you'll see this log message: >

Saved offset is before Replication slot's confirmed_flush_lsn, Airbyte will trigger sync from scratch

This happens when the WAL segments needed for the incremental sync have already been removed by Postgres (e.g., due to high write volume or infrequent syncing). [Postgres troubleshooting] Controlling the Behavior The Invalid CDC Position Behavior (Advanced setting) controls what happens when the saved offset is invalid: • Fail sync: The sync fails with an error message. • Re-sync data: Airbyte triggers a full refresh from scratch silently. [CDC position behavior] General Resumability Airbyte is a deliver at-least-once system — it will retry and may re-deliver records already sent. For CDC streams specifically, resumability depends on the WAL offset still being available in Postgres. If you want to avoid duplicate records at the destination, use a sync mode that includes deduplication. [Resumability docs] In summary: For CDC, Airbyte attempts to resume from the last committed WAL offset, but will fall back to a full re-sync if that offset is no longer available in the Postgres replication slot. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai How can I increase the number of failure retries?

Based on the conversation context (Postgres → BigQuery CDC sync), there are two different levels where you can configure retry behavior: 1. Platform-Level Job Retries (OSS/Self-Hosted) For self-hosted Airbyte, you can configure retry limits via environment variables: •

SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_SUCCESSIVE

– Max successive attempts where no data was synchronized •

SYNC_JOB_RETRIES_COMPLETE_FAILURES_MAX_TOTAL

– Max total attempts where no data was synchronized •

SYNC_JOB_RETRIES_PARTIAL_FAILURES_MAX_TOTAL

– Max total attempts where some data was synchronized By default, Airbyte allows: • 5 successive attempts with no data synced • 10 total attempts with no data synced • 20 total attempts with some data synced [Configuring Airbyte Jobs] [Jobs & Retries] 2. Connector-Level HTTP Retries (Config-Based Connectors) If you're building or modifying a connector using the YAML/config-based framework, you can increase the HTTP request retry count via the

error_handler

in the `Requester`:

requester:
  error_handler:
    max_retries: 10  # default is 5

[Error Handling YAML] Note: For Airbyte Cloud, the platform-level retry environment variables are not user-configurable. The knowledge sources indicate that connection-level or workspace-level retry controls are a requested feature but not yet available as a built-in option. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Where is the "backoff" setting controlled?

Based on the conversation context, there are two levels where backoff is controlled: 1. Platform-Level Job Retry Backoff (OSS/Self-Hosted) For sync job retries, backoff between failed attempts is configured via these environment variables: •

SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_MIN_INTERVAL_S

– Minimum backoff interval in seconds between failed attempts where no data was synchronized •

SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_MAX_INTERVAL_S

– Maximum backoff interval in seconds •

SYNC_JOB_RETRIES_COMPLETE_FAILURES_BACKOFF_BASE

– Exponential base of the backoff interval By default, Airbyte backs off with: • 10 seconds after the 1st complete failure • 30 seconds after the 2nd • 90 seconds after the 3rd • 4 minutes 30 seconds after the 4th [Configuring Airbyte Jobs] [Jobs & Retries] 2. Connector-Level HTTP Request Backoff (Config-Based Connectors) For HTTP request retries within a connector, backoff is configured in the connector's YAML via the `error_handler`'s

backoff_strategies

. The supported strategies are: •

ExponentialBackoffStrategy

– Default; uses a

factor

(default: 5) with interval

factor * 2^attempt_count

•

ConstantBackoffStrategy

– Fixed wait via

backoff_time_in_seconds

•

WaitTimeFromHeader

– Reads wait time from a response header •

WaitUntilTimeFromHeader

– Waits until a time specified in a response header [Error Handling YAML] [Connector Builder Error Handling] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.