How does deleted fields work? I thought airbyte would handle it fine with "propagate field changes only" by automatically adjusting the schema by removing the removed field at source

How Airbyte Handles Deleted/Removed Fields The behavior here is a bit nuanced and has been a source of confusion. Here's what the documentation says vs. what has been observed in practice: Documented/Intended Behavior According to Airbyte's schema change management documentation, when a column/field is removed from the source: • The old field and its historical data are retained in the destination — the column is not dropped. • Subsequent rows are written with

NULL

for that removed field. • The field and its historical data will only be removed entirely if the connection is ever cleared or refreshed. [Schema Change Management] So with "Propagate field changes only", Airbyte is not supposed to drop the column from the destination — it should keep it and just write

NULL

going forward. What "Propagate Field Changes Only" Actually Does This setting propagates column additions and changes, but for removals, the intended behavior is retention (not deletion). It does not mean the column will be automatically dropped from the destination. [Schema Change Management] Known Issues & Exceptions There have been real-world reports of columns being unexpectedly deleted from the destination despite the documented "retain" behavior: • A GitHub issue reported that with Postgres → Snowflake in CDC mode, a renamed or deleted column was completely removed from Snowflake, causing data loss — contradicting the documented behavior. [GitHub Issue #54719] • A documentation PR was opened specifically to align the Snowflake connector docs with the platform-level contract, noting that the connector was incorrectly issuing

DROP COLUMN

on propagated field removals. [GitHub PR #82720] ClickHouse Exception If you're using ClickHouse as a destination, the behavior is explicitly different — it does drop removed columns and their historical data, due to how its ReplacingMergeTree engine works. [ClickHouse Schema Evolution] Summary If you're seeing columns being dropped unexpectedly (especially with Snowflake), this may be a connector-level bug that has been acknowledged and is being addressed. It's worth checking your connector version and whether a fix has been released. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai how does this change if you're using postgres cdc

Deleted Fields with Postgres CDC Using Postgres CDC adds some important complications on top of the standard behavior: CDC-Specific Limitations The Postgres source documentation explicitly states: > "Schema changes are not supported automatically for CDC sources. Reset and resync data if you make a schema change." This means that even with "Propagate field changes only" enabled, CDC does not handle schema changes as cleanly as standard replication methods. [Postgres Troubleshooting] Why CDC Makes This Worse With CDC (WAL-based replication), schema changes like column deletions are particularly problematic because: • Changes made via

ALTER

statements do not appear in the WAL logs and therefore won't be captured by CDC. Only `DELETE`/`INSERT`/`UPDATE` operations are captured. [Postgres Troubleshooting] • When a column is removed at the source, CDC events for subsequent rows simply won't include that field — but Airbyte may not cleanly handle the transition. Observed Real-World Behavior A confirmed issue report showed that with Postgres CDC → Snowflake using "Propagate field changes only": • A deleted (or renamed) column was completely dropped from Snowflake, rather than being retained with

NULL

values. • This happened for both column renames and outright deletions. • The behavior contradicted the documented "retain" contract. [GitHub Issue #54719] Recommended Approach for CDC + Schema Changes Given these limitations, the safest approach when dealing with column deletions in a CDC pipeline is: 1. Manually refresh the source schema in the Airbyte UI after making schema changes in Postgres. 2. Reset and resync the affected streams to ensure consistency between source and destination. 3. Avoid relying on automatic propagation for schema changes in CDC mode — treat it as a manual operation. [Postgres Troubleshooting] In short, CDC mode makes deleted field handling less reliable and less predictable than with standard replication, and the safest path is a manual reset/resync rather than depending on automatic schema propagation. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai the doc says "Schema changes are not supported automatically for CDC sources. Reset and resync data if you make a schema change." Does this mean even new fields are not flowing through to the table(s) with "propagate field changes only"?

New Fields in Postgres CDC with "Propagate Field Changes Only" This is an important distinction. The documentation statement about schema changes not being supported automatically for CDC is specifically about the CDC transport mechanism (WAL logs), not necessarily about Airbyte's schema propagation feature. What the CDC Limitation Actually Means The core issue is that

ALTER TABLE

statements (like adding a column) don't appear in WAL logs. So CDC cannot detect the schema change through the replication stream itself. [Postgres Troubleshooting] New Columns Can Flow Through, But With a Catch The Airbyte CDC documentation describes a specific process for adding new columns: [CDC Limitations] 1. Airbyte detects the new column via its schema check (not via CDC logs) 2. You'll see a Schema changes detected notice in the Schema section 3. You must review and approve the new column 4. Then trigger a Refresh Stream on the affected stream to snapshot it before CDC tracking begins > "If you create a new schema/column that is not yet being synced, it must first be snapshotted before CDC can begin tracking changes." The Key Risk: Backfill Before Schema Refresh There is a known open issue where if you: 1. Add a column 2. Backfill existing rows with

UPDATE

3. Then refresh the schema in Airbyte ...the backfill

UPDATE

events captured by CDC are lost because Airbyte didn't know about the column yet when they occurred. Existing rows end up with

NULL

in the new column until a full refresh is performed. [GitHub Issue #80920] Recommended Order of Operations for New Columns in CDC 1.

ALTER TABLE ... ADD COLUMN ...

2. Immediately trigger a schema refresh in Airbyte and approve the new column 3. Then perform any backfill

UPDATE

statements This ensures CDC events from the backfill are captured correctly with the new column already known to Airbyte. Summary So to directly answer your question: new fields can flow through, but they require more manual intervention in CDC mode than with standard replication, and the order of operations matters significantly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai Can you "refresh source schema" via the api, not in the ui?

Refreshing Source Schema via API Yes, there is an API endpoint for this. Based on the knowledge sources, you can trigger a schema discovery (equivalent to "Refresh source schema") via:

POST /v1/sources/discover_schema

[Forum Post] This endpoint runs the

discover

step on the connector, which fetches the latest schema from the source. Important Caveats However, the knowledge sources do not contain enough detail about the full request/response shape of this endpoint, or whether it automatically applies the discovered schema changes to the connection (vs. just returning the new schema for review). Based on older forum discussions, there were historical issues where schema updates via API did not persist to the connection — the schema would revert after a reset/resync. [Forum Post] However, this may have changed in newer versions of Airbyte. Recommended Next Steps For the most accurate and up-to-date details on this endpoint, I'd recommend checking: • The Airbyte Public API reference directly • The Airbyte API docs for your deployment type (Cloud vs. self-managed) Given the CDC-specific concerns we discussed earlier, even if the API call succeeds, you'd still want to follow the correct order of operations (refresh schema before backfilling) to avoid data loss. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai Is it possible to apply the schema changes via api too? e.g. do a discovery, and apply the changes

Applying Schema Changes via API The knowledge sources do not contain enough detailed information to give a fully confident answer here, but based on what is available: What the API Supports The

PATCH /connections/{connectionId}

endpoint allows you to update connection settings, including

nonBreakingSchemaUpdatesBehavior

, which controls how Airbyte handles detected schema changes. [API Reference] The older internal API also had a

POST /v1/connections/update

endpoint, but there is a confirmed historical issue where schema changes sent via this endpoint did not persist — they would revert after a reset or resync. [Forum Post] The Core Limitation A support agent explicitly confirmed: > "You can update the connection, but only the schema won't persist." > "Running the discover schema endpoint will run the

discover

step in the connector. You can't overwrite the values generated by this process." [Forum Post] What You Can Do Programmatically The most reliable programmatic approach appears to be: 1. Set your connection to

propagate_fully

or

propagate_columns

via

PATCH /connections/{connectionId}

using the

nonBreakingSchemaUpdatesBehavior

field — this lets Airbyte auto-apply non-breaking changes when detected. [API Reference] 2. Trigger a sync via

POST /v1/jobs

with

jobType: sync

— Airbyte checks for schema changes before each sync and will auto-propagate them if the setting above is configured. What's Missing The knowledge sources don't confirm whether there is a dedicated public API endpoint to: • Trigger a schema discovery and atomically apply the results to a connection in one call I'd recommend checking the Airbyte API reference directly or reaching out via the Airbyte Community Slack for the most up-to-date answer on this. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.