@kapa.ai, i have airbyte 1.7.0 i have setup incremental append pipeline using cdc method my source is postgres and my destination is s3 but for some reason it is not replication incremental records to s3

Here are the most common causes and fixes for Postgres CDC incremental syncs succeeding but transferring 0 records to S3: 1. Insufficient Initial CDC Waiting Time (Most Likely Cause) The connector may be timing out while searching for the WAL resume position. The default wait time is 20 minutes (1200 seconds). If the connector can't find the WAL position in time, it closes the connection without reading any records. You can identify this by checking your sync logs for something like:

Searching for WAL resume position
Source has no more messages, closing connection.

with a ~5 minute gap between them (indicating the wait time was exceeded). [Postgres troubleshooting] Fix: Increase the "Initial Waiting Time in Seconds (Advanced)" setting on the connector. The valid range is 120–2400 seconds. Try setting it to 600 first to test, and up to 2400 if needed. [Postgres troubleshooting] 2. WAL Files Have Been Purged If the WAL logs needed for the incremental sync have been removed by Postgres, the connector may fall back to a full refresh or sync 0 records. This can happen when: • There are too many DB updates filling the

pg_wal

directory. • The connector acknowledged WAL records, but the destination failed to consume them. You'll see this log message: >

Saved offset is before Replication slot's confirmed_flush_lsn, Airbyte will trigger sync from scratch

Fix: Sync more frequently, or increase

wal_keep_size

on your Postgres instance. [Postgres troubleshooting] 3. Replication Slot Not Being Consumed If the replication slot is accumulating data but not being flushed, it could indicate the connector is not properly reading from the WAL. [Forum post] Fix: Check that your replication slot and publication are correctly set up. Also ensure the replication slot was created after the initial table population to avoid duplicate records. [CDC tutorial] ------------------------- Recommended first step: Check your sync logs for the WAL resume position timeout, and try increasing the Initial Waiting Time in Seconds to 2400 in the advanced connector settings. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai getting this error in logs

IntegrationRunner$Companion(stopOrphanedThreads):442                  The main thread is exiting while children non-daemon threads from a connector are still active.
                 Ideally, this situation should not happen...
                 Please check with maintainers if the connector or library code should safely clean up its threads before quitting instead.
                 The main thread is: main (RUNNABLE)
Thread stacktrace: java.base/java.lang.Thread.getStackTrace(Thread.java:2450)

The warning

The main thread is exiting while children non-daemon threads from a connector are still active

is a known issue in Airbyte connectors. Here's what the knowledge sources say about it: What This Warning Means This is a warning (not a hard error) indicating that some background threads in the connector didn't finish cleaning up before the main thread exited. Ideally, connectors should safely clean up their threads before quitting. [Forum post] Impact on Your Sync Based on reported cases: • The sync may complete successfully but with 0 records or missing records — which aligns with the issue you described (no incremental records being replicated to S3). [Forum post] • In some cases, it can also corrupt the CDC cursor state, causing subsequent syncs to break. [Github Issue] What You Can Try 1. Reset the connection and perform a full resync — This can help re-establish a clean CDC state if the cursor has been corrupted. 2. Change the "Invalid CDC position behavior (Advanced)" setting from

Fail sync

to

Re-sync

. This allows the connector to perform a full refresh when the CDC position is invalid, after which incremental behavior should resume. [Github Issue] 3. Check your connector version — Some versions of the Postgres source connector have known bugs with CDC state management. Version

3.6.15

and

3.6.18

have been reported as more stable compared to later versions like

3.6.28

and

3.7.0

. [Github Issue] 4. Increase the "Initial Waiting Time in Seconds (Advanced)" — As mentioned in the previous response, a timeout while searching for the WAL resume position can cause the connector to exit prematurely, which may trigger this warning. This warning is a known issue that the Airbyte team is aware of, but the knowledge sources do not contain a definitive root cause fix specific to the Postgres → S3 CDC setup in your version (1.7.0). 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai, what should be

wal_keep_size ?

Based on the knowledge sources, here are the recommendations for `wal_keep_size`: Recommended

wal_keep_size

Configuration The knowledge sources don't specify a single exact value, but provide these guidelines: • WAL retention should ideally be 7 days worth of WAL data — this covers weekend maintenance and most data movement scenarios. [CDC best practices] • At a minimum, WAL retention must be longer than your sync frequency to prevent LSN loss between syncs. [CDC best practices] Examples by sync frequency: | Sync Frequency | Minimum Retention | Recommended | | - | - | - | | Every 6 hours | 1–2 days | 7 days | | Daily | 2–3 days | 7 days | | Every 3 days | 4–5 days | 7 days | Practical values seen in the community: • One user successfully used

wal_keep_size = 2048

MB (2 GB) but later increased to

3072

MB (3 GB) for stability. [Forum post] • The AlloyDB issue reporter had

wal_keep_size = 100 GB

and

max_slot_wal_keep_size = 800 GB

. [Github Issue] The right value depends on your transaction volume and sync frequency. The key principle is: ensure WAL is retained long enough that it won't be purged between two consecutive syncs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I am using aws rds and this is my settings for wal_keep_size

max_slot_wal_keep_size
-
Dynamic
Integer

Based on the knowledge sources, for AWS RDS, the relevant WAL-related parameters are: •

wal_keep_size

— Sets the size of WAL files held for standby servers. •

max_slot_wal_keep_size

— Sets the maximum WAL size that can be reserved by replication slots. From a community user running Postgres CDC on RDS with a similar setup, these were the working parameter values: [Forum post] | Parameter | Value | | - | - | |

max_slot_wal_keep_size

| 3072 MB (increased from 2048 MB) | |

wal_keep_size

| 2048 MB | |

max_wal_size

| 2048 MB | > That user was syncing hourly and found that 2048 MB for

max_slot_wal_keep_size

was not enough overnight when no data changes occurred, so they increased it to 3072 MB. [Forum post] Key Recommendation The right value depends on your transaction volume and sync frequency. The general principle from Airbyte's CDC best practices is: [CDC best practices] • Optimal: 7 days worth of WAL retention • Minimum: WAL retention must be longer than your sync frequency Since

max_slot_wal_keep_size

on RDS is a dynamic integer parameter (as you noted), you can adjust it without a reboot. A starting point of 3072–4096 MB is reasonable for most setups, but if your database has high transaction volume, you may need significantly more. Also note that for AWS RDS, you need to enable

pg_cron

via the parameter group if you want to set up a heartbeat table to keep the replication slot active during low-activity periods. [Postgres troubleshooting] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.