@kapa.ai I have a connector to postgres RDS with 5 tables that sync to Snowflake using CDC. 4 of the tables are 2B rows large. Each table takes longer than 24hours to sync on it's own for an initial sync, meaning a timeout will occur (since they occur every 24 hours). Airbyte seems to be missing a lot of data when it syncs during this initial load. Once it starts incrementing after the initial snapshot, things seem to work 100%. However, we're missing a lot of data (millions of rows) from the initial sync. My initial waiting time is 2400 seconds Size of queue = 10K LSN commit behaviour is "After loading data in the destination". Initial load timeout in hours = 60. Do you have any tips for me to increase the success rate of our initial snapshot? We use airbyte version 2.0.1 and snowflake connector 4.0.27

I’m sorry, but the knowledge sources don’t contain a clear, prescriptive recipe for reliably completing very long (>24h) Postgres CDC initial snapshots of multi‑billion‑row tables; they mostly describe related limitations and tuning levers. I’ll summarize what is documented and how you might use it. ### 1. Make sure WAL / CDC position isn’t the bottleneck For large CDC workloads, Airbyte may time out while trying to find or resume from the WAL position. In one reported case, increasing Initial Waiting Time in Seconds (Advanced) to 2400 seconds resolved “successful syncs with 0 records” for incremental tables, together with increasing WAL size. [[PG CDC wait](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time); [Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)] You already set this to 2400s, which is at the upper end of the documented range (120–1200s in older docs, but support explicitly recommended 2400s in the issue). [[PG CDC wait](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time); [Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)] If you haven’t already: - Ensure WAL retention / size on RDS is large enough so WALs are not purged while the long snapshot is running. In the referenced case, users increased WAL size to 1 TB (though they weren’t sure it was strictly required). [[Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)] ### 2. Tune snapshot / initial load behavior The “Initial Load Timeout” parameter controls how long Airbyte will stay in snapshot mode before switching to CDC streaming. For very large databases, docs recommend 12–24 hours, with a hard maximum of 24h in that doc set. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] Your value (60h) is above what’s documented there; I don’t have information in the sources about behavior beyond 24h or whether that’s fully supported in your version. Because of that, I can’t say confidently whether this is contributing to missed data, only that it’s outside the documented guidance. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] Given the docs: - Monitor the first sync and set Initial Load Timeout to something close to the observed snapshot duration, but not far beyond 24h if your Airbyte build enforces that limit. [[CDC best practices](https://docs.airbyte.com/platform/understanding-airbyte/cdc-best-practices#source-configuration)] ### 3. Reduce back‑pressure from Snowflake For CDC sources with many tables, slow flushing on the Snowflake destination can cause back‑pressure and very low throughput. A support answer for Postgres→Snowflake CDC recommends: [[PG→Snowflake CDC](https://discuss.airbyte.io/t/4373)] - Use staging on the Snowflake destination. - Increase the Snowflake “buffer” parameter (they suggest 50) to improve throughput. - Limit the number of tables per CDC connection (they recommend ≤30 tables per connection; you’re at 5, so that’s fine). [[PG→Snowflake CDC](https://discuss.airbyte.io/t/4373)] In that case, simply increasing the Snowflake buffer to 50 solved severe slowness. [[PG→Snowflake CDC](https://discuss.airbyte.io/t/4373)] Given your tables are huge and initial loads are very long, any throughput gain on the destination side directly helps reduce the window where snapshot + WAL overlap can cause issues. ### 4. General scaling / performance considerations For very large initial loads (hundreds of millions to billions of rows), Airbyte support has suggested: - Scaling Kubernetes / worker parallelism to increase throughput, especially for CDC initial loads that risk overrunning log retention. [[MSSQL CDC slow](https://discuss.airbyte.io/t/1686)] However, the docs don’t provide concrete CPU/memory numbers or exact tuning steps; they only suggest “play around with your kubernetes configuration and try to increase the job parallelism” and refer to general worker parallelization docs (not included in the provided sources). [[MSSQL CDC slow](https://discuss.airbyte.io/t/1686)] ### 5. Known Postgres CDC edge cases There are known issues around: - Wrong LSN committed during initial snapshot: if the job fails before some CDC changes are processed, the saved offset can be invalid, leading to failures or re‑syncs depending on “Invalid CDC position behavior”. [[Issue 49803](https://github.com/airbytehq/airbyte/issues/49803)] - Suggested mitigations in that issue include: - Setting Invalid CDC position behavior to “Re‑sync Data” so the pipeline retries snapshot instead of failing permanently. [[Issue 49803](https://github.com/airbytehq/airbyte/issues/49803)] - Temporarily adding high‑traffic tables to the publication so that changes are more likely to occur after snapshot and correct the broken offset. [[Issue 49803](https://github.com/airbytehq/airbyte/issues/49803)] The sources don’t explicitly tie this to “missing millions of rows in the initial snapshot”, but if your initial job fails or restarts during snapshot, this behavior could interact badly with long‑running loads. ### 6. What the docs do not clearly cover The provided sources do not clearly explain: - How to guarantee a complete initial snapshot when it takes >24h. - How Airbyte behaves when Initial Load Timeout is set above the documented 24h maximum. - A supported way to “chunk” or “resumably” snapshot a single huge Postgres table (there is a user discussion about resumable full refresh, but no official answer yet). [[CDC resumable discussion](https://github.com/airbytehq/airbyte/discussions/52058)] Because of that, I can’t confidently recommend a specific “resumable snapshot” configuration for your case. --- ### Practical steps you can try, within what’s documented Within the limits of the docs above, the most grounded actions are: 1. Ensure WAL retention is large enough (RDS parameter group) so WALs are not purged during the entire initial snapshot + catch‑up period. [[Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)] 2. Keep Initial Waiting Time high (you already use 2400s) so the connector has enough time to find the resume position. [[PG CDC wait](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-setting-up-initial-cdc-waiting-time); [Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)] 3. Optimize Snowflake destination throughput: - Enable staging. - Increase buffer parameter to around 50. [[PG→Snowflake CDC](https://discuss.airbyte.io/t/4373)] 4. Scale Airbyte workers / K8s resources to increase read/write throughput for the initial snapshot. [[MSSQL CDC slow](https://discuss.airbyte.io/t/1686)] 5. Consider Invalid CDC position behavior = “Re‑sync Data” if you see failures during snapshot and want Airbyte to keep retrying instead of failing permanently, understanding this can increase load. [[Issue 49803](https://github.com/airbytehq/airbyte/issues/49803)] Given the gaps in the docs, if you still see missing rows after these changes, the next step would likely be to open a detailed issue or support ticket with Airbyte including logs and exact connector configs, since the behavior you’re seeing (multi‑billion‑row tables, >24h snapshots, missing millions of rows) goes beyond what’s explicitly covered in the available documentation.