When using postgres data source, when using this option Refresh your data Refreshing your data will replace all data in your destination by reloading all data from the source. -- what are these choices, and what are the consequences of each? Refresh streams and retain records By electing to refresh your streams and retain records, you are instructing Airbyte to keep any records in your destination that the source no longer contains. This option is recommended for analytics use cases or where downstream reporting depends on historical data. Refresh streams and remove records By electing to refresh your streams and remove records, you are instructing Airbyte to remove previously synced records that are no longer in the source. This option is recommended for database replication use cases where you expect a 1:1 mirror of the source. Use this option carefully as this is not reversible.

Refresh Options in Airbyte (Postgres Source) When you trigger a Refresh Sync in Airbyte, it erases the current cursor for incremental streams and re-syncs data from the beginning. You are presented with two choices: 1. Refresh Streams and Retain Records This option keeps any records already in your destination and merges in the newly synced data. Records that existed in the destination but are no longer present in the source will not be removed. Consequences: • Old and new copies of your data are both stored, leading to increased data storage in your destination. • May result in increased compute costs and slower syncs, depending on your destination. • Best used when you need to recover from an unreliable (e.g., lossy or buggy) source, especially with a Deduplicated sync mode. • Recommended for analytics use cases or where downstream reporting depends on historical data. • The

_airbyte_generation_id

column increments, allowing you to distinguish records from before and after the refresh. [Refresh & Retain Records] 2. Refresh Streams and Remove Records This option only keeps data that was newly synced since the refresh was initiated. Any records previously in the destination that no longer exist in the source will be removed. Consequences: • Produces a clean, up-to-date copy of the source data in your destination — a 1:1 mirror. • Irreversible — if your source does not retain all of its historical records, this may lead to data loss. • If you need to preserve history, you should back up your destination data before running this option. • Recommended for database replication use cases where you want the destination to exactly mirror the source. [Refresh & Remove Records] Key Points Applicable to Both Options • No data downtime: While the refresh is running, the previous data remains queryable. The new data is only swapped in upon successful completion — unlike the older "Reset Sync" behavior. [Refreshing your data] • Scheduled syncs are blocked until the refresh completes. • Streams not being refreshed will continue to sync normally during the refresh. • Both refresh types are resumable if the source and destination support checkpointing. • The

_airbyte_generation_id

increments with every refresh, helping you track which generation of data each record belongs to. [Data Generations] > Note: For the Postgres source specifically, availability of these options depends on the sync mode configured for each stream (e.g., "Refresh and Remove Records" is only available for Incremental | Append + Dedupe streams, not for Full Refresh modes). [Refreshing your data] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

does one take longer than the other?

Based on the knowledge sources, there is no direct comparison of the execution time between the two options. However, there are some relevant hints: • Refresh and Retain Records is noted to lead to increased compute costs and slower syncs, because both the old and new copies of your data are stored and must be processed together. [Refresh & Retain Records] • Refresh and Remove Records is described as the "simplest type of Refresh Sync," which implies it is comparatively more straightforward to execute. [Refreshing your data] This suggests that Refresh and Retain Records will generally take longer and cost more than Refresh and Remove Records, due to the overhead of merging old and new data. However, the knowledge sources don't provide specific timing benchmarks or a definitive side-by-side comparison, so I'd recommend treating this as a general guideline rather than a guarantee. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

How to log a bug with postgres source? We use incremental and the connector missed two records that were inserted into the DB at almost the exact same time the airbyte connection occured. Wanted someone to help us fix this, we had to do a full refresh to get those missed records into our destination

To report this as a bug with the Postgres source connector, you should open a GitHub issue following Airbyte’s standard process for data replication bugs. 1. Where to log the bug Airbyte asks users to report replication bugs in the main GitHub repo: 1. First, search existing issues to avoid duplicates: https://github.com/airbytehq/airbyte/issues?q=is%3Aissue+is%3Aopen+label%3Atype%2Fbug [Issues & requests] 2. If you don’t find your case, open a new issue using the bug template: https://github.com/airbytehq/airbyte/issues/new/choose [Issues & requests] For your case, choose the data replication bug template and specify source-postgres as the connector. 2. What details to include (especially for “missed records”) From similar Postgres issues in the tracker, the following details make it much easier for the team to investigate: • Connector & Airbyte versions ◦ Postgres source connector version (e.g.

3.7.0

,

3.6.18

, etc.) [CTID bug; CTID cursor reset] ◦ Airbyte OSS / Cloud and its version. • Sync configuration ◦ Sync mode: confirm it’s Incremental and whether it’s: ▪︎ Standard cursor / CTID-based incremental, or ▪︎ CDC (Debezium-based) incremental. [CDC error; CDC LSN bug] ◦ Destination type and version. ◦ Cursor field (e.g.

updated_at

,

id

) and primary key. ◦ Any advanced Postgres source settings (e.g. xmin vs CDC, replication slot, publication). • Reproduction details ◦ Approximate time window when: ▪︎ The two records were inserted in Postgres. ▪︎ The Airbyte incremental sync ran. ◦ Whether those rows: ▪︎ Appeared in the source DB before and after the sync. ▪︎ Were missing in the destination until you ran a full refresh. ◦ Whether the incremental sync run was marked successful in Airbyte UI (no errors). Similar “silent miss” behavior is documented in other issues. [CTID bug] • Logs ◦ Attach the full logs for: ▪︎ The incremental sync that missed the records. ▪︎ The full refresh that later picked them up. ◦ This is exactly what maintainers ask for in similar “records missed” reports. [JSON not copied] • Table / workload characteristics ◦ Table size and write pattern (e.g. “high write rate, multiple inserts per second”). ◦ Whether there are concurrent updates to the same rows during sync (relevant to CTID-based scanning issues). [CTID bug] 3. Getting help / follow-up • After you open the issue, it will be triaged (often by an AI assistant plus a human maintainer) and, if needed, escalated internally, as you can see in similar Postgres issues. [CTID bug; CTID cursor reset] • For more interactive help, you can also: ◦ Join the Airbyte Community Slack and share the issue link for discussion. ◦ If you’re an Airbyte Cloud customer, open a support ticket and reference your GitHub issue. [Issues & requests] The knowledge sources don’t give a specific root cause for “two inserts at the exact sync time” in your scenario, but they do show that similar “missed records” and CTID/CDC edge cases are actively investigated once reported with detailed logs and configuration. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai what should I write in the relevant information section, can you write it for me?

Here’s a template you can paste into the “Relevant information” section of a GitHub issue for the Postgres source, adapted from how other Postgres bugs are reported in the tracker [CTID cursor reset; CTID scan bug]:

**Summary**

During an incremental sync from Postgres, the connector missed two records that were inserted into the source database at almost the exact time the Airbyte sync started. The sync completed successfully with no errors, but those two records did not appear in the destination until we ran a full refresh.

**Environment**

- Airbyte deployment: (Cloud or OSS; version X.Y.Z)
- Source: Postgres
    - Connector: source-postgres (version X.Y.Z)
    - Postgres version: (e.g., 13.10, 15.12, etc.)
- Destination: (e.g., Snowflake / BigQuery / Postgres; connector version X.Y.Z)

**Sync configuration**

- Sync mode: Incremental
- Destination sync mode: (Append / Append + Deduped)
- Replication method:
    - [ ] Standard (CTID / cursor-based, non-CDC)
    - [ ] CDC (Debezium-based)
- Cursor field: `<your cursor column, e.g. updated_at or id>`
- Primary key: `<pk column(s)>`
- Any advanced Postgres source settings:
    - (e.g., xmin vs CDC, replication slot name, publication name, etc.)

**Table / workload characteristics**

- Approx. row count: `<N rows>`
- Write pattern: records are inserted/updated in batches; in this case, two inserts happened very close to the sync start time.
- Are there frequent concurrent writes/updates during sync? (Yes/No; brief description)

**Steps to reproduce**

1. Configure a connection from Postgres to `<destination>` with:
      - Incremental sync on table `<table_name>`
      - Cursor field `<cursor_column>`
2. Run an incremental sync at time **T0**.
3. At approximately **T0** (same minute/second), insert two new rows into `<table_name>`.
4. Let the incremental sync complete (status is **Succeeded** in Airbyte).
5. Check the destination:
      - The two new rows are **present in the source** but **missing in the destination**.
6. Trigger a **Full Refresh** on the same connection.
7. After the full refresh completes, the previously missing rows appear in the destination.

**Expected behavior**

All rows that exist in the source at or before the end of the incremental sync window should be replicated to the destination. Rows inserted around the sync time should not be permanently missed; they should either appear in that sync or in a subsequent incremental sync, without requiring a manual full refresh.

**Actual behavior**

- The incremental sync completed successfully with no errors.
- Two rows inserted at (or very close to) the sync start time were not replicated.
- Subsequent incremental syncs did not pick them up.
- Only a manual full refresh caused those rows to appear in the destination.

**Evidence / verification**

- Verified in Postgres that the two rows existed before and after the incremental sync.
- Verified in the destination that:
    - They were missing after the incremental sync.
    - They appeared only after the full refresh.
- No obvious errors or warnings in the Airbyte logs for the incremental sync.

**Logs**

- Attached logs for:
    - The incremental sync that missed the records.
    - The full refresh that later picked them up.
(Please see attached log files.)

You can fill in the placeholders (versions, table name, cursor column, timestamps) with your actual details before submitting. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.