Hi @kapa.ai, I've recently deployed a new airbyte instance after successfully migrating our postgres db. This is part of our effort to upgrade out airbyte from 0.5.3 to 1.8.5. While the deployment is successful and a new test connection runs perfectly, there is a big problem: Whenever I re-enable a sync from the previous deployment, even though they run incremental refreshes, airbyte will immediately start to clear the data instead of running the normal incremental refresh. While this is not a huge issue for our smaller tables, we have big tables that can take 2 entire days to fully reset. How can I switch off this auto-clearing of data?

Thinking completed I am sorry, but the provided knowledge sources do not mention a specific toggle or configuration setting to entirely switch off this automatic data clearing behavior when reenabling connections after a major migration. However, the knowledge sources show that this is a known issue reported during major version upgrades (such as migrating from older versions to 1.5+): Why this happens • When migrating across major Airbyte versions, existing connection manager workflows or connector state schemas can become mismatched or missing. • In logged bug reports [Github Issue: Airbyte triggers data clears; Github Issue: source-mysql Data loss], instances that had disabled/paused connections before upgrading logged:Failed to retrieve ConnectionManagerWorkflow for connection [connectionID]. Repairing state by creating new workflow.During this state/workflow repair or upon detecting schema differences on reactivation, Airbyte erroneously enqueued a stream reset/clear job automatically, even when settings like

Approve all changes myself

were selected [Github Issue: source-mysql Data loss]. • Additionally, major upgrades often trigger schema change checks. If Airbyte detects breaking schema changes (such as cursor or primary key differences) or stream definitions that do not match the expected state format, it defaults to resetting or clearing the streams before resuming [Github Issue: source-postgres Clear/reset blanks; Schema change management]. Things you can check 1. Check Schema Change Management Settings: Navigate to the connection’s Settings tab and check your schema update policy. Setting it to Approve all changes myself or Propagate field changes only ensures Airbyte does not automatically adjust or wipe streams for non-breaking changes [Schema change management]. Note: Community users reported that the reactivation bug still triggered in some versions despite this setting [Github Issue: source-mysql Data loss]. 2. Verify Schema and Cursors Manually First: Before re-enabling or running a sync, go to the connection's Schema tab, click Refresh source schema, ensure the incremental cursor field and primary keys are properly mapped and saved, and decline any prompt to reset the stream if one appears [Connector upgrade guide; Schema change management]. 3. Check the Connection State: Major upgrades across massive version gaps (such as 0.5.x to 1.8.x) significantly changed connector internals and state structures (e.g., transitions to per-stream state or CDK changes). If the source detects incompatible state formats, clearing may be triggered or required to establish a valid sync state [Github Issue: source-postgres Clear/reset blanks]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Hmm @kapa.ai, I changed to

Propagate all field and stream changes"

and it looks like it's working? The connection is running syncs instead of clearing the data

Thinking completed That makes sense based on how Airbyte handles schema changes. When a connection is set to Propagate all field and stream changes, Airbyte automatically detects new streams, stream removals, column additions/deletions, and data type changes, and applies them directly into the ensuing sync without pausing or prompting for manual intervention (unless a breaking change, like a removed primary key or cursor, is detected) [Detect and Propagate Schema Changes connection setting]. By contrast, when manual approval or other propagation modes are active, or when changing propagation settings between manual and automated modes, Airbyte often flags differences as pending schema changes. In those states—or when manually reviewing and saving schema changes—Airbyte frequently marks streams as affected and prompts or queues a clear/reset before allowing syncs to resume [Github Issue (open): Data Loss: Changing Schema Propagation Settings Causes Silent Dropping of CDC Tables; Detect and Propagate Schema Changes connection setting]. Important cautions moving forward: • Avoid changing propagation settings mid-flight: Reported issues show that changing the propagation setting itself (e.g., from Propagate all field and stream changes to Propagate field changes only) can sometimes be treated by Airbyte as a schema change that triggers table recreation or clears [Github Issue (open): Data Loss: Changing Schema Propagation Settings Causes Silent Dropping of CDC Tables]. • Breaking changes still pause: If a primary key or cursor column is ever dropped or renamed at the source, Airbyte will still pause the connection regardless of this setting, and saving the updated cursor/primary key will typically prompt you to clear the stream [Detect and Propagate Schema Changes connection setting]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@Liew Tze Hao Timothy we also had this same issue, in our case it was really odd - any syncs that were triggered manually post-upgrade were fine, but ones that were scheduled and had not been run manually post-upgrade did the clear, i think we had propagate column changes only / manually approve for our settings anyways not much help but hope the process wasn't too traumatic (it was really traumatic for us 😄)

@Alasdair Ellis Oh I see, strangely (but thankfully) changing the schema change management to the auto-propagate modes did the trick for our case, and I guess it was more tedious than traumatic for us 😅 Still, good to know we're not the only ones that ran into this problem!