https://linen.dev logo
Join Slack
Powered by
When a field in a table deleted at source, how doe...
# replication-ask-ai
y
When a field in a table deleted at source, how does airbyte hande that? Is it possible to not error out but still raising a warning saying the field has been deleted or something like that (maybe imputing with null values)
k
How Airbyte Handles a Deleted Source Field Default Behavior When a column/field is removed from the source, Airbyte retains the column and its historical data in the destination — it does not drop the column. For all subsequent rows synced after the deletion, Airbyte writes 
NULL
for that field. [Schema Change Management] So by default, you already get the behavior you're describing: no hard error, and 
NULL
imputation for the missing field going forward. > ⚠️ Note: If the connection is ever cleared or refreshed, the retained column and its historical data will be removed entirely from the destination. [Schema Change Management] Is It a Breaking Change? A removed column/field is not classified as a breaking change by Airbyte. Only the removal of a cursor or primary key is considered breaking (which pauses the connection immediately). [Schema Change Management] Getting Notified (Warning Behavior) You can configure the connection to detect and notify you of schema changes without automatically propagating them: • "Approve all changes myself" — Airbyte detects the change and shows a blue "i" icon on the Schema tab, but continues syncing with the old schema. You manually review and approve changes. [Schema Change Management] • "Be notified when schema changes occur" — There is also a notification setting on the connection's Settings tab to receive alerts when schema changes are detected. [Schema Change Management]

docs.airbyte.com/assets/images/schema-propagation-options…▾

Summary So your desired behavior — no error, 
NULL
imputation, and a warning — is essentially what Airbyte provides out of the box, especially when combined with the "Approve all changes myself" or notification settings. image 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai I set my config "approve all change myself" but the sync failed saying there was no longer a column in the table. Why is that then?
k
This appears to be a known bug. There is an open GitHub issue reporting exactly this behavior — where connections set to "Approve all changes myself" still fail (or silently error) when a column is removed from the source, instead of continuing to sync the remaining columns as expected. [GitHub Issue] The error message you're likely seeing looks something like:
Copy code
Field 'column_name' not found in stream 'your_stream'.
To continue syncing this stream, restore access to the field
or refresh the source schema to remove it.
What's happening The expected behavior under 
Approve all changes myself
is: • Removed columns should not block the sync • Remaining columns should continue syncing • Changes should only be applied when you manually approve them However, in practice, some connector versions (notably source-postgres v3.8.0-rc.11 was cited) have been observed to fail the sync entirely when a column is removed, rather than gracefully continuing. This was escalated to Airbyte's engineering team for triage. [GitHub Issue] Workaround Until the bug is resolved, the recommended workaround is to manually refresh the source schema and approve the removal of the deleted column: 1. Go to Connections → select your connection → Schema tab 2. Click Refresh source schema 3. Review the detected changes and click Save changes This will update Airbyte's catalog to reflect the column removal, and subsequent syncs should no longer fail on that missing field. [Schema Change Management] If you're an Airbyte Cloud customer, it's worth opening a support ticket referencing the issue above. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai yeah I'm using postgres connector with cdc. So this error is expected and everything you mentioned stays true?
k
Good clarification — with CDC mode, there are some additional caveats worth knowing. CDC + Schema Changes The Postgres documentation explicitly states: > Schema changes are not supported automatically for CDC sources. Reset and resync data if you make a schema change. [Postgres Troubleshooting] This means that with CDC, a removed column is more likely to cause sync failures than in non-CDC modes, because CDC relies on the WAL (Write-Ahead Log) and changes made via 
ALTER
(like dropping a column) won't appear in the logs and can cause mismatches between what Airbyte expects and what the source provides. [CDC Limitations] Does "Approve all changes myself" still apply? The schema change management settings still apply, but the recommended approach for CDC is actually to use 
Approve all changes myself
specifically to prevent unreviewed changes from being applied without a proper snapshot first. [CDC Limitations] However, the bug described in the previous answer — where syncs fail on column removal even under 
Approve all changes myself
— still applies on top of this. [GitHub Issue] Recommended Steps for CDC + Removed Column 1. Refresh the source schema in the Airbyte UI (Schema tab → Refresh source schema) 2. Review and approve the column removal 3. Trigger a stream refresh on the affected stream to re-snapshot it 4. Resume syncing This is the safest path to avoid further failures when schema changes occur in a CDC-enabled Postgres source. [Postgres Troubleshooting] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai that makes me think auto change schemas seem like tjr safer approach for cdc in airbyte? What are other options are there for postgres cdc?
k
Actually, it's the opposite — auto schema propagation is riskier for CDC, not safer. Here's why and what your options are: Why Auto Propagation is Risky for CDC There is a documented case where changing schema propagation settings on a CDC connection caused complete data loss — Airbyte silently dropped and recreated all changelog tables without any warning. [GitHub Issue] Additionally, with CDC, new columns/streams must be snapshotted first before CDC can begin tracking changes on them. If auto-propagation picks up a new schema change and includes it in a sync without a proper snapshot, it can cause LSN (Log Sequence Number) issues and sync failures. [CDC Limitations] Available Options for Postgres CDC Airbyte's Official Recommendation for CDC > We recommend enabling 
Approve all changes myself
under Detect and propagate schema changes to prevent newly added detected changes from being included in the sync without a proper snapshot, reducing the risk of LSN issues and sync failures. [CDC Limitations] So despite the bug you encountered, "Approve all changes myself" is still the safest option for CDC — it just currently has a known issue with column removals that requires a manual schema refresh as a workaround. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai that means you just need to know exactly when a field is removed right? And the schema evolution within airbyte is useful for that with cdc
k
Yes, exactly. The core challenge with CDC + column removal is knowing when it happens so you can act before the next sync fails. Airbyte's schema change management is useful for this purpose. How It Helps With "Approve all changes myself" configured, when a column is dropped at the source, Airbyte will: 1. Detect the schema change before or during the next sync 2. Surface it visually — a blue "i" icon appears on the Schema tab [Schema Change Management] 3. Notify you if you have schema change notifications enabled [Schema Change Management] This gives you the opportunity to: • Review the change • Manually refresh the source schema • Trigger a stream refresh before resuming syncs The Limitation to Be Aware Of The knowledge sources do not explicitly confirm that schema detection is guaranteed to happen before the sync fails in CDC mode. As seen in the bug report, the sync can sometimes fail on the missing column before you get a chance to review and approve the change. [GitHub Issue] Practical Takeaway Schema change management with 
Approve all changes myself
is your best available tool for staying informed about field removals in CDC, but it's not a perfect safety net yet. Pairing it with: • Schema change notifications (to get alerted quickly) • Frequent sync schedules (to consume CDC logs regularly) [CDC Limitations] ...gives you the best chance of catching and handling column removals before they cause problems. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai is there a way to send a notification only on a schema change when a field is removed on certain tables?
k
I'm sorry, but the knowledge sources do not contain enough information to confirm that level of granularity in Airbyte's notification system. What Airbyte's Notifications Currently Support Airbyte's schema change notifications work at the connection level, not at the table/stream or field level. You can enable: • Connection Updates — notified when any schema change is automatically applied • Connection Updates Requiring Action — notified when a breaking change requires manual intervention These are toggled per workspace and per connection, but there is no mention of filtering notifications by: • Specific tables/streams • Specific type of change (e.g. only field removals) [Manage Notifications] What You Could Do Today The closest workaround would be: • Set "Approve all changes myself" on connections for the tables you care about • Enable schema change notifications for those specific connections • Manually review each detected change in the Schema tab But you would still receive notifications for all schema changes on that connection, not just field removals on specific tables. Recommendation If this is important to your use case, I'd suggest: 1. Checking the Airbyte Community Slack for any workarounds 2. Opening a feature request on the Airbyte GitHub — there is already a related discussion around more granular schema change controls [GitHub Discussion] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai I don't like the whole connector fails when there is a schema change in one table that is not critical. "Propagate field changes only" or another one that propagates changes automatically is my best bet here? With notification turned on
k
That's a reasonable approach, but there are important trade-offs to understand for CDC specifically. "Propagate field changes only" for CDC This option would automatically apply column removals without pausing the connection, which solves your problem of the whole connector failing on a non-critical table change. With notifications enabled, you'd still be alerted when changes occur. [Schema Change Management] However, the key risks for CDC are: • New columns added at source will be auto-propagated without a proper snapshot first, which can cause LSN issues and sync failures [CDC Limitations] • There is a documented case where changing the propagation setting itself (e.g. switching from "Propagate all" to "Propagate field changes only") caused silent data loss on CDC changelog tables [GitHub Issue] "Propagate all field and stream changes" This carries even more risk for CDC — new streams would also be auto-applied without snapshots, compounding the LSN issues. [CDC Limitations] Honest Assessment The knowledge sources don't provide a clean "safe" option for your exact scenario with CDC. The trade-off is essentially: So "Propagate field changes only" + notifications may reduce your sync failures, but you'd be trading that for increased CDC snapshot risks on column additions. It's worth weighing how frequently columns are added vs removed in your case. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai Why does connector fail on removeal
k
Based on the knowledge sources, the exact root cause is not fully documented, but here is what is known: Why the Connector Fails on Column Removal The failure appears to be a bug in the source-postgres connector. Under 
Approve all changes myself
, the expected behavior is that syncs continue with the old schema — but instead, the connector throws a hard error like:
Copy code
Field 'column_name' not found in stream 'your_stream'.
To continue syncing this stream, restore access to the field
or refresh the source schema to remove it.
[GitHub Issue] Why It's Worse with CDC With CDC specifically, there is a mismatch between: • Airbyte's catalog — which still expects the old column to exist • The WAL logs — which no longer contain that column since it was dropped via 
ALTER TABLE
Since `ALTER`/`TRUNCATE` changes do not appear in the WAL logs, Airbyte's CDC layer can get confused when it encounters records that don't match the schema it has stored in its catalog. [Postgres Troubleshooting] What's Missing The knowledge sources do not go deeper into the internal connector code to explain exactly why the connector raises a hard failure rather than gracefully handling the mismatch. This would likely require reviewing the connector source code or waiting for an official response on the open GitHub issue. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai Usually when there is a removed field, until you let it through it in the UI, it won't affect anything, right? But because I'm using cdc postgres, this errors out
k
Yes, that's exactly the right way to think about it. Normal Behavior (Non-CDC) Under 
Approve all changes myself
, a removed field is detected but held for your review. The sync continues using the old catalog — the removed column just gets 
NULL
values — and nothing breaks until you explicitly approve the change in the UI. [Schema Change Management] Why CDC Breaks This Expectation With CDC + Postgres, the connector is reading from the WAL logs rather than querying the table directly. When a column is dropped via `ALTER TABLE`: • The change does not appear in the WAL logs [Postgres Troubleshooting] • Airbyte's catalog still expects the column to exist • When the CDC layer processes log events and can't reconcile the catalog against what it's reading, it hard fails rather than gracefully continuing So the fundamental issue is that CDC's reliance on WAL logs makes it much less tolerant of schema mismatches compared to standard replication, where Airbyte can simply query the table and handle missing columns gracefully. In Short The knowledge sources don't provide full details on whether this is purely a bug or partially expected CDC behavior, but the practical outcome is the same — CDC makes the "hold until approved" safety net much less reliable. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai What has the community come up with to put a patch on this?
k
Based on the knowledge sources, the community discussions are mostly focused on the CDC LSN/WAL issues rather than specifically the schema change + CDC failure. Here is what has been suggested: Community Workarounds 1. Manual schema refresh before syncing The most commonly cited workaround is to manually refresh the source schema and approve the column removal in the UI before the next sync runs, preventing the catalog mismatch from occurring. [GitHub Issue] 2. Switch "Invalid CDC position behavior" to "Re-sync data" Rather than 
Fail sync
, setting this to 
Re-sync data
means the connector will attempt a full re-snapshot instead of hard failing. The trade-off is cost and time for large tables. [GitHub Issue] 3. Downgrade the connector version Some community members found that downgrading to source-postgres v3.6.18 or v3.6.15 avoided several CDC-related failures, though this is not specific to schema changes and comes with its own trade-offs of missing newer fixes. [GitHub Issue] Honest Caveat The knowledge sources do not contain a community-proposed patch specifically for the schema change + CDC column removal failure bug. The GitHub issue was escalated to Airbyte's engineering team but no community-developed fix is documented. [GitHub Issue] Your best bet for an up-to-date community discussion would be the Airbyte Community Slack or monitoring the open GitHub issue directly for engineering updates. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai Educate me on pros and cons on using "propagate field changes only" on postgres cdc connectors again ?
k
Based on our conversation and the knowledge sources, here is a focused breakdown: "Propagate Field Changes Only" on Postgres CDC ✅ Pros • Avoids the column removal hard failure — instead of erroring out, column changes are automatically applied, reducing sync disruptions on non-critical tables [Schema Change Management] • Stream changes are ignored — new or removed streams won't be auto-applied, giving you some control over larger structural changes [Schema Change Management] • Notifications still work — you can stay informed of changes without the sync blocking on them ❌ Cons • New columns auto-applied without a snapshot — with CDC, any new column detected will be propagated without a proper initial snapshot first, which can cause LSN issues and sync failures [CDC Limitations] • Data loss risk on column removal — there is a documented case where auto-propagating column changes on a CDC connection caused silent data loss [GitHub Issue] • Changing this setting itself is risky — switching to or from this setting on an existing CDC connection has been reported to trigger silent table drops and data loss [GitHub Issue] • Historical data not retained — unlike 
Approve all changes myself
where removed columns retain 
NULL
values, auto-propagation may remove the column entirely from the destination [Schema Change Management] Bottom Line It trades sync stability for data safety, which for CDC is generally the wrong trade-off. The knowledge sources consistently point to 
Approve all changes myself
as the recommended setting for CDC, despite its current bug with column removals. [CDC Limitations] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai If I decided to use a propagation approach, would it make sense to use "back fill new or renamed column" option as well?
k
Good question, but in the context of CDC + "Propagate field changes only", enabling backfill for new or renamed columns has some important nuances. What "Backfill New or Renamed Columns" Does When a new or renamed column is detected through auto-propagation, Airbyte re-syncs the entire stream so all rows get populated with values for the new column, not just newly updated rows. [Backfill Setting] Does It Make Sense with CDC? Probably not, for a few reasons: 1. It only works with auto-propagation The backfill only triggers when schema changes are detected as part of an automatic sync — it does not trigger if you manually refresh the schema. So it does pair correctly with 
Propagate field changes only
in that sense. [Backfill Setting] 2. CDC already struggles with new columns without snapshots As discussed, new columns in CDC need to be snapshotted first before CDC can track them. The backfill option essentially forces a full stream re-sync, which is similar to a snapshot — but the knowledge sources do not confirm this is handled safely in CDC mode. [CDC Limitations] 3. Increased destination costs Backfilling re-syncs the entire stream every time a new or renamed column is detected, which can be expensive for large tables. [Backfill Setting] 4. Compounds existing CDC risks Given that 
Propagate field changes only
already carries data loss and LSN risks on CDC, adding automatic full stream re-syncs on top of that increases the surface area for something going wrong. Bottom Line The knowledge sources do not explicitly address this combination for CDC, so I'd be cautious about recommending it confidently. If you do go this route, testing on a non-critical stream first would be advisable before applying it broadly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai where do you set your wal retention period?
k
Where to Set WAL Retention Period The WAL retention period is configured at the database level, not in Airbyte. As the docs explicitly state: > "This is configured in your database, not in Airbyte." [WAL Retention] How to Configure It in Postgres There are two main Postgres parameters depending on your version: Postgres 13+
Copy code
-- Set WAL retention size (in MB, 0 = disabled by default)
ALTER SYSTEM SET wal_keep_size = '<size_in_mb>';
SELECT pg_reload_conf();
Postgres < 13
Copy code
-- Set number of WAL segments to retain
ALTER SYSTEM SET wal_keep_segments = '<number_of_segments>';
SELECT pg_reload_conf();
[Postgres Troubleshooting] What Value Should You Set? Per Airbyte's recommendations: For example: • Syncing every 6 hours → minimum 1-2 days (7 days recommended) • Syncing daily → minimum 2-3 days (7 days recommended) • Syncing every 3 days → minimum 4-5 days (7 days recommended) [WAL Retention] The 7-day recommendation provides the best buffer against unexpected delays or maintenance windows. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai if the docs is says: "If your connection setting is set to 
Propagate field changes only
or 
Propagate all field and stream changes
, schema changes except for breaking changes are automatically applied to the ensuing sync and the data in the destination will automatically shift as schema changes are applied." So removing fields is a breaking change that doesn't get auto propagated?
k
That's a really sharp observation, but the answer is no — removing a field is not classified as a breaking change by Airbyte. What Airbyte Considers a Breaking Change According to the docs, only these two scenarios are breaking changes: • An existing primary key is removed from the source • An existing cursor is removed from the source [Schema Change Management] A regular column removal is not in that list. So What Does That Mean? Under 
Propagate field changes only
, a removed column should be auto-propagated — meaning: • The column is removed from the destination • Historical data in that column is lost • No manual approval required This is actually consistent with the data loss bug reported in the GitHub issue, where a column removal under propagation settings caused the column to be completely dropped from Snowflake rather than retained. [GitHub Issue] The Contradiction You're Sensing The confusion likely comes from the fact that: • The docs say removed columns are retained with 
NULL
values [Schema Change Management] • But in practice with auto-propagation, they appear to get dropped entirely This contradiction was actually acknowledged in a recent Airbyte PR that attempted to align the documentation, noting the "retain" behavior was the planned/intended behavior but the connector code wasn't fully matching it yet. [GitHub PR] So in short — field removal is not a breaking change, it will be auto-propagated, and the actual destination behavior is still being stabilized. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai For whateve reason, cdc is not working and you want to refresh all your data, can you just use the "Refresh your data" option?
k
Yes, you can use the "Refresh your data" option, but there are important considerations specific to CDC. What a Refresh Does A Refresh Sync erases the current cursor/state and re-syncs all data from the beginning — effectively re-snapshotting your tables. Crucially: • No data downtime — previous data remains queryable until the refresh completes [Refreshing Your Data] • Blocks scheduled syncs until complete [Refreshing Your Data] • Resumable if the source and destination support checkpointing [Refreshing Your Data] Which Refresh Option to Choose for CDC? Since CDC typically uses Incremental | Append + Dedupe, you have two options: [Refreshing Your Data] For CDC recovery specifically, "Refresh and retain records" is the safer choice since: > "Using a Refresh and Retain Records Sync with a Deduplicated Sync mode is the best way to recover from an unreliable (e.g. lossy or buggy) source." [Refresh and Retain Records] One CDC-Specific Caveat If your CDC failure is related to the orphaned stream state bug (where removed streams leave stale state behind), a refresh alone may not fully reset the shared CDC state, meaning the next sync could still fail with the same LSN error. In that case, a full connection clear via the API may be needed. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai does doing "refreshs tream" per table - does that also do a full sync?
k
Yes, Refresh Stream per table does trigger a full re-sync for that specific stream. What "Refresh Stream" Does From the docs: > "Refresh stream re-syncs all historical data for the stream." [Per-Stream Actions] It is essentially the same as the connection-level refresh but scoped to a single stream/table, meaning: • Only that stream's cursor/state is reset • All historical data is re-pulled from the source for that table • Other streams in the connection are unaffected and continue syncing normally [Refreshing Your Data] For CDC Specifically This is actually the recommended approach when adding new columns or schemas in CDC mode: > "For new schemas/columns: Locate the newly added schema, click the three dots (⋮) next to it, and choose Refresh Stream." [CDC Limitations] This makes per-stream refresh particularly useful for CDC because: • You avoid re-snapshotting all tables when only one is problematic • Each stream has an independent generation ID, so refreshing one won't affect others [Refreshing Your Data] • It's less costly and disruptive than a full connection refresh 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai does that mean doing the whole "Refresh stream" on the connector itself is the same thing by doing "refresh stream" per table one by one?
k
Yes, the end result is essentially the same — but there are some practical differences worth noting. Connection-Level Refresh vs Per-Stream Refresh [Refreshing Your Data] [Per-Stream Actions] Key Difference The important distinction is that a connection-level refresh blocks all scheduled syncs until it completes, whereas doing it per stream allows other streams to continue syncing normally in the meantime. [Refreshing Your Data] So while doing per-stream refreshes one by one would eventually achieve the same data outcome as a full connection refresh, it is: • ✅ Less disruptive to other streams • ✅ Lower data latency across the connection • ❌ More manual effort if you have many tables For CDC recovery specifically, doing it per stream is generally the better approach unless every single table is broken and needs re-snapshotting simultaneously. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements
y
@kapa.ai is there a way to do per stream refresh and stop other tables syncs? The thing is our wal log / replicatin slot got wiped and we need to refresh everything from scratch
k
Given your situation — where the WAL log/replication slot got wiped — doing per-stream refreshes while keeping other streams syncing actually doesn't make sense, and a connection-level refresh is the more appropriate approach. Why Connection-Level Refresh Makes More Sense Here When the replication slot is wiped/lost: • All streams are affected — CDC state is shared across all streams via the global shared state (LSN position) [GitHub Issue] • Per-stream refreshes only reset individual stream states, but the shared CDC state (LSN) remains broken for all streams • Other streams continuing to sync while the slot is wiped will likely fail anyway So there's no real benefit to keeping other streams syncing in this scenario. Recommended Approach 1. Pause the connection to stop any scheduled syncs from running and failing 2. Recreate the replication slot in Postgres if it was dropped:
Copy code
SELECT pg_create_logical_replication_slot('airbyte_slot', 'pgoutput');
3. Run a full connection-level refresh — this resets the shared CDC state and re-snapshots all tables [Refreshing Your Data] One Additional Caveat If your connection has orphaned stream states (streams previously removed from the catalog), the connection-level refresh may not fully reset the shared CDC state, leaving the stale LSN in place. In that case you'd need to manually overwrite the state via the API:
Copy code
POST /api/v1/state/create_or_update
{
  "connectionId": "...",
  "connectionState": {
    "stateType": "global",
    "connectionId": "...",
    "globalState": { "shared_state": {}, "streamStates": [] }
  }
}
[GitHub Issue] This is the most reliable way to fully reset the CDC state when the replication slot has been wiped. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai And even if the full refreshing is taking place, thte normal sync (every hour for example) won't take place until it's done
k
Yes, that is correct. The docs explicitly state: > "Refresh Syncs will block any scheduled syncs from occurring until they are complete." [Refreshing Your Data] What Happens During a Full Refresh • All scheduled syncs are blocked until the refresh completes • The previous data remains queryable in the destination — no data downtime [Refreshing Your Data] • Once the refresh completes successfully, the newly refreshed data is swapped into place and normal scheduled syncs resume One Silver Lining Even though scheduled syncs are blocked, the docs note that: > "During a Refresh Sync, the streams that are not being refreshed will also sync normally in order to lower data latency." [Refreshing Your Data] However, in your specific case where the replication slot is wiped and you're doing a full connection-level refresh, all streams are being refreshed — so this silver lining doesn't really apply. Every stream is blocked until the full re-snapshot completes. This is another reason why for large datasets, a full connection refresh after a replication slot wipe can be particularly disruptive — your hourly syncs won't run until every table has been fully re-snapshotted. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai so if we want to fully refresh data for the cdc connection, you need to wipe out the state too? If you do "Refresh stream" you ignore wal logs, correct? We're using postgre
k
Great question, and this is an important distinction for Postgres CDC specifically. Refresh Stream vs State Reset The knowledge sources do not explicitly confirm whether "Refresh Stream" automatically wipes the shared CDC state (LSN) in Postgres CDC. This is an important gap to flag. However, based on what we know: "Refresh Stream" resets the per-stream state — it re-snapshots the table from scratch. But as discussed earlier, CDC in Postgres uses a global shared state containing the LSN position that is shared across all streams. [GitHub Issue] So the concern is: • ✅ Per-stream state is reset • ⚠️ Shared CDC state (LSN) may not be wiped — meaning after the snapshot, Airbyte could still try to resume from the old WAL position What This Means Practically For a full CDC recovery on Postgres, the safest approach based on the knowledge sources is: 1. Wipe the shared state manually via the API to clear the stale LSN:
Copy code
POST /api/v1/state/create_or_update
{
  "connectionId": "...",
  "connectionState": {
    "stateType": "global",
    "connectionId": "...",
    "globalState": { "shared_state": {}, "streamStates": [] }
  }
}
[GitHub Issue] 2. Then run a full connection-level refresh to re-snapshot all tables On Ignoring WAL Logs The knowledge sources do not explicitly confirm that "Refresh Stream" bypasses WAL logs entirely. What is known is that: • The initial snapshot in CDC is always a full 
SELECT
— effectively a Full Refresh that doesn't use WAL [CDC Overview] • Subsequent syncs after the snapshot resume from the WAL position stored in the shared state So if the shared state LSN is not cleared, even after a refresh snapshot completes, the next incremental sync would still attempt to resume from the old WAL position — which could fail if the replication slot was wiped. This is why wiping the state is critical in your scenario. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai how do you wipe the states? Can I just remove everything in the connection state? I have access via the ui
k
Yes, you can do it via the UI, but with important caveats. How to Edit State via the UI 1. Go to Connections → select your connection 2. Click the Settings tab 3. Click the Advanced dropdown 4. Click anywhere in the Connection State to start editing 5. Clear the state and replace with an empty global state 6. Click "Update state" to confirm [Modifying Connection State]

docs.airbyte.com/assets/images/airbyte_connection_update_state…▾

What to Set It To Rather than removing everything entirely, set it to an empty global state:
Copy code
{
  "stateType": "global",
  "globalState": {
    "shared_state": {},
    "streamStates": []
  }
}
[GitHub Issue] ⚠️ Important Warnings > "Updates to connection state should be handled with extreme care. Updates may break your syncs, requiring a full historical sync of your data to fix. Make changes only as directed by the Airbyte team." [Modifying Connection State] A few things to be aware of: • The UI state editor has a known issue where setting a stream state to 
null
can cause a schema validation error on the next sync [GitHub Issue] • Setting 
streamStates
to 
[]
(empty array) rather than nulling individual streams is the safer approach • After wiping the state, immediately run a full connection refresh to re-snapshot all tables before the next scheduled sync kicks in image 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai currently the state is set like this ""[\"prod-api\",{\"server\":\"prod-api\"}]": "{\"lsn_proc\":568767733426536,\"messageType\":\"DELETE\",\"lsn_commit\":568767733409952,\"lsn\":568767733426536,\"txId\":3942970431,\"ts_usec\":1786147274645245}"" And in each stream no state info is stored. I suppose this is because this is cdc?
k
Yes, exactly — this is completely normal and expected for a CDC connection. What You're Seeing Your state looks like this:
Copy code
{
  "shared_state": {
    "state": {
      "[\"prod-api\",{\"server\":\"prod-api\"}]": "{\"lsn_proc\":568767733426536,\"messageType\":\"DELETE\",\"lsn_commit\":568767733409952,\"lsn\":568767733426536,\"txId\":3942970431,\"ts_usec\":1786147274645245}"
    }
  },
  "streamStates": []
}
This is the Debezium offset stored as the global shared state — it contains: • 
lsn
 — the Log Sequence Number Airbyte last processed • 
lsn_proc
 — the LSN of the last processed event • 
lsn_commit
 — the LSN of the last committed transaction • 
txId
 — the transaction ID • 
ts_usec
 — timestamp in microseconds • 
messageType
 — the last CDC event type (in your case a 
DELETE
) [Postgres CDC Operations] Why Stream States Are Empty This is because CDC uses a Global state type where: • The shared state (LSN position) tracks where Airbyte is in the WAL log — shared across all streams • Individual stream states are largely unnecessary since all streams are tracked via the single WAL position [Airbyte Protocol State Types] To Wipe This for a Full Refresh Based on your earlier scenario of needing a full re-snapshot, you'd want to replace this entire state with:
Copy code
{
  "stateType": "global",
  "globalState": {
    "shared_state": {},
    "streamStates": []
  }
}
This clears the LSN position, forcing Airbyte to treat the next sync as a cold start and re-snapshot all tables from scratch. [GitHub Issue] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai how do you tell it to start from the beggining of this week?
k
The knowledge sources do not contain enough information to confirm whether Airbyte supports setting a specific date/time as the CDC starting point for Postgres. What Is Known For non-CDC incremental streams, you can edit the cursor value in the connection state to a specific date, effectively telling Airbyte to start syncing from that point. [Modifying Connection State] However, for Postgres CDC specifically, the state is driven by the LSN position (as seen in your current state), not a timestamp — even though the state does contain a 
ts_usec
field. The Challenge To start from "the beginning of this week" in CDC terms, you would theoretically need to: 1. Find the LSN value that corresponds to the start of this week in your Postgres WAL logs 2. Manually set that LSN in the shared state But the knowledge sources do not confirm whether: • Airbyte supports manually setting a specific LSN as a starting point • The 
ts_usec
field in the shared state can be used to control where CDC resumes from Practical Recommendation This is a question best directed to: • Airbyte Community Slack for community input • Airbyte Support if you're on Cloud As this goes beyond what the available documentation covers for CDC state manipulation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai If I had the shared state like this how do you clear this? Or what would be the "from scratch" json value I can put it? "shared_state": { "state": { "[\"prod-api\",{\"server\":\"prod-api\"}]": "{\"lsn_proc\":569049015797616,\"messageType\":\"UPDATE\",\"lsn_commit\":569049015367272,\"lsn\":569049015797616,\"txId\":3946386177,\"ts_usec\":1786153642582757}" } }, "streamStates": [ {...
k
Based on our earlier conversation and the knowledge sources, to clear this and start from scratch you would replace the entire state with this in the UI:
Copy code
{
  "stateType": "global",
  "globalState": {
    "shared_state": {},
    "streamStates": []
  }
}
[GitHub Issue] What This Does • 
shared_state: {}
 — clears the Debezium offset (your LSN position), forcing a cold start • 
streamStates: []
 — clears all stream states, including any orphaned ones that could prevent the shared state from being properly reset Steps in the UI 1. Go to Connections → your connection 2. Click Settings → Advanced 3. Click the Connection State to edit 4. Replace everything with the JSON above 5. Click "Update state" ⚠️ Important After clearing the state: • Immediately run a full connection refresh before the next scheduled sync kicks in • The next sync will treat this as a cold start and re-snapshot all tables from scratch • Do not let the hourly scheduled sync trigger before the refresh — it could write a new broken state [Modifying Connection State] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai My format is like this though?
Copy code
{
  "shared_state": {
    "state": {
      "[\"prod-api\",{\"server\":\"prod-api\"}]": "{\"lsn_proc\":569049015797616,\"messageType\":\"UPDATE\",\"lsn_commit\":569049015367272,\"lsn\":569049015797616,\"txId\":3946386177,\"ts_usec\":1786153642582757}"
    }
  },
  "streamStates": [
    {
      "streamDescriptor": {
        "name": "changes",
        "namespace": "billing_audit"
      },
      "streamState": {}
    },
    {
      "streamDescriptor": {
        "name": "account_roles",
        "namespace": "public"
      },
      "streamState": {}
k
Good catch — your state format is slightly different from what the UI expects as input. Based on the knowledge sources, the full JSON you'd need to put in the UI to clear it from scratch would be:
Copy code
{
  "stateType": "global",
  "globalState": {
    "shared_state": {},
    "streamStates": []
  }
}
[GitHub Issue] Why the Format Looks Different What you're seeing in the UI is the raw stored state format, but when you update the state via the UI, it expects the Airbyte protocol wrapper format with: • 
stateType: "global"
— tells Airbyte this is a global CDC state • 
globalState
— wraps both 
shared_state
and 
streamStates
What Happens to Your Existing Streams By setting 
streamStates: []
you are clearing all your existing stream states too, including: • 
billing_audit.changes
• 
public.account_roles
• Any other streams in your connection This is intentional in your case since you want a full re-snapshot from scratch. After updating the state, remember to: 1. Pause scheduled syncs immediately 2. Run a full connection refresh before the next hourly sync triggers [Modifying Connection State] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
y
@kapa.ai Is it possbiel via the api, you can do the per-stream full refresh on multiple streams at once? Doing it one by one takes time and inefficient (we don't want to do the full refresh on the whole connector)
k
Based on the knowledge sources, there is unfortunately not enough information to confirm whether the public API supports triggering a per-stream refresh on multiple streams simultaneously. What Is Known The public 
POST /v1/jobs
endpoint currently only supports: • 
sync
• 
reset
And explicitly rejects 
jobType=REFRESH
with 
"Refreshes are not supported in the public API"
— meaning even a single stream refresh via the public API is not currently supported. [GitHub Issue] The Gap This is actually an open issue that has been escalated to Airbyte's engineering team. A community member noted that the internal machinery for refreshes already exists and is what the UI calls — but it's simply not wired to the public API yet. [GitHub Issue] Your Options Right Now 1. UI only — do per-stream refreshes manually one by one via the UI 2. Full connection refresh — if doing them one by one is too inefficient and you need all streams refreshed anyway, a full connection-level refresh may be the pragmatic choice despite being more disruptive 3. Monitor the GitHub issue — there is active discussion about adding 
REFRESH
support to the public API, including multi-stream support [GitHub Issue] Recommendation Given your scenario where the replication slot was wiped and you need everything refreshed anyway, a full connection refresh after clearing the shared state is likely your most practical path forward right now, rather than waiting for per-stream API support to be implemented. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
6 Views
Open in Slack
PreviousNext
Powered by