<@U04S8PLJWMT> how do we prevent airbyte from dele...
# ask-ai
j
@kapa.ai how do we prevent airbyte from deleting fields when MySQL CDC logs a field has been removed
k
Airbyte does not physically drop columns from the destination when the source schema removes a field. Instead: - When a column/field is removed in the source, **the existing column remains in the destination but stops being updated**; it only disappears if you clear/reset the connection or stream. [[Schema mgmt 1.8](https://docs.airbyte.com/platform/1.8/using-airbyte/schema-change-management); [Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] - This behavior applies regardless of CDC; CDC only affects how row changes are read (`DELETE`/`INSERT`/`UPDATE` from the binlog), not the schema-change policy. [[CDC 1.6](https://docs.airbyte.com/platform/1.6/understanding-airbyte/cdc)] So, if your goal is to prevent Airbyte from deleting or dropping fields in the destination when MySQL CDC logs that a column was removed: 1. Do not reset/clear the connection or affected streams after the schema change — that is what would remove the old column and its historical data. [[Schema mgmt 1.8](https://docs.airbyte.com/platform/1.8/using-airbyte/schema-change-management)] 2. In the connection’s Schema change management setting, choose one of: - Propagate field changes only – column additions/deletions are recognized, but removed columns stay in the destination and just stop updating. [[Detect & propagate – platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management#detect-and-propagate-schema-changes-connection-setting)] - Approve all changes myself – Airbyte continues syncing with your last saved schema; you can choose not to apply the removal. [[Detect & propagate – next](https://docs.airbyte.com/platform/next/using-airbyte/schema-change-management#detect-and-propagate-schema-changes-connection-setting)] Based on the docs, there is **no setting that would make Airbyte automatically drop destination columns on source removal**; the default behavior already preserves them as you want. If you are seeing columns actually being dropped, the knowledge sources do not describe such behavior, so I can’t explain it from the docs alone.
j
@kapa.ai it looks like airbyte deleted the fields, we are using Incremental | Append + Dedupe and Propogate field changes only
k
A message from kapa.ai
Based on the docs, what should be happening with your settings is: - When a column/field is removed in the source, Airbyte **keeps the existing column in the destination but stops updating it**; it is only removed if the connection/stream is cleared or refreshed. [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] - With Propagate field changes only, “only column changes will be propagated. New or removed streams will be ignored.” [[Detect & propagate – platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management#detect-and-propagate-schema-changes-connection-setting)] - Incremental | Append + Deduped affects row behavior (deduplication on PK + cursor), not column dropping. [[Incremental deduped](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped)] So according to the documentation: 1. **A removed field in MySQL should not cause its column to be dropped in the destination.** It should remain, just no longer receive updated values. [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] 2. The column and its historical data would be removed only if the connection/stream was cleared/reset or fully refreshed (for example, a “clear” or “refresh” operation of the stream/connection). [[Schema mgmt 1.8](https://docs.airbyte.com/platform/1.8/using-airbyte/schema-change-management)] The knowledge sources you provided do not describe any behavior where: - MySQL CDC + “Incremental | Append + Deduped” + “Propagate field changes only” - automatically drops destination columns when the field is removed in the source. Because of that, I can’t explain your observed column deletion from the docs alone. The most the docs say is that removal of a column in the source should result in: > “The old field will be retained in the destination, but stop updating with updated values. If the connection is ever cleared or refreshed, the field and its historical data will be removed entirely.” [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] If you’re certain no reset/clear/refresh was done and columns still disappeared, that behavior is not documented in the sources you shared, so I can’t definitively account for it. In that case, you’d likely need to: - Check Airbyte job logs for any reset/clear/refresh events on that connection/stream. - Consider opening a support ticket / GitHub issue with details, since the observed behavior diverges from the documented expectations.
j
@kapa.ai Timeline only shows Sync Succeeded events and a Schema Updated event
k
A message from kapa.ai
With the configuration you described: - Sync mode: Incremental | Append + Deduped - Schema change setting: Propagate field changes only the documented behavior for a removed column/field is: - “The old field will be retained in the destination, but stop updating with updated values. If the connection is ever cleared or refreshed, the field and its historical data will be removed entirely.” [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management); [Schema mgmt 1.8](https://docs.airbyte.com/platform/1.8/using-airbyte/schema-change-management)] And “Propagate field changes only” means only column changes are propagated; it does not say that a destination column will be dropped, just that the change is detected and applied to how data is synced. [[Detect & propagate – platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management#detect-and-propagate-schema-changes-connection-setting)] The knowledge sources you provided do not describe any case where: - a Schema Updated event by itself, plus - Sync Succeeded jobs would cause columns to be physically deleted from the destination without an explicit clear/refresh/reset of the stream/connection. Given that: - If you see columns actually removed from the destination and the timeline only shows Sync Succeeded and Schema Updated (no reset/clear/refresh), that specific behavior is not documented in the sources we have. - The docs explicitly state that removal of a column in the source should retain the column in the destination until a clear/refresh. [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] So, from the documentation alone I cannot explain why the fields disappeared. The next concrete steps, beyond what the docs say, would be to: 1. Double‑check that no “clear”/“reset”/“refresh stream” was triggered around the time of the Schema Updated event, since that is what the docs associate with actual removal of historical columns/data. [[Schema mgmt platform](https://docs.airbyte.com/platform/using-airbyte/schema-change-management)] 2. If you confirm none were, this looks inconsistent with the documented behavior and should be raised via Airbyte support or a GitHub issue, including: - connector (MySQL CDC) and destination types/versions, - connection ID, - timestamps of the Schema Updated and subsequent syncs, - and before/after DDL of the affected table in the destination. From the available knowledge sources, I can’t be more specific than that.