Hi everyone 👋 We've been debugging a silent CDC failure on Airbyte 2.1.0 (self-hosted, GKE) and found what looks like a platform bug. Sharing here in case others are affected or someone knows of a fix. 🐛 Bug: nightly discover resets CDC state by updating

sourceCatalogId

even on empty schema diff Setup • Airbyte 2.1.0, self-hosted on GKE •

source-mysql:3.51.5

(CDC mode,

invalid_cdc_cursor_position_behavior: Fail sync

) •

destination-bigquery:3.0.16

•

nonBreakingChangesPreference: ignore

What happens Every night at midnight UTC a scheduled discover runs. When it completes,

ConnectionsHandler.applySchemaChange()

updates the connection's

sourceCatalogId

— even when the schema diff is empty. New syncs launched under the new

catalogId

find no associated CDC state and fall back to an old saved binlog position. The result is a silent, growing backlog: each sync reprocesses an ever-increasing number of records from the old binlog position. In our case this grew from ~2M to ~55M records over 2 days before BigQuery started returning 500s and the sync loop became permanent. Why

nonBreakingChangesPreference: ignore

doesn't help This setting controls stream-level propagation, but

applySchemaChange()

still updates

sourceCatalogId

on every discover completion regardless. The CDC state orphaning happens upstream of that logic. Impact • Pipeline silently accumulates backlog for days without alerting • Failure only surfaces when destination (BigQuery) starts rejecting large batch inserts • By then the binlog position in the CDC state may no longer exist (rotated out) Question Is this a known issue? Is there a way to prevent

sourceCatalogId

from being updated when there's no actual schema change? We couldn't find a config option that controls this behavior. Current mitigation We deployed a Cloud Run job (hourly) that queries sync stats and alerts on Slack if any CDC connection shows 5 consecutive runs with monotonically increasing records emitted where the latest run processes >2x the records of the oldest in the window. This lets us catch the backlog early before volumes become critical. Thanks 🙏

Hi, I'm interested in understanding what's going on here.

New syncs launched under the new

catalogId

find no associated CDC state and fall back to an old saved binlog position.

The catalog ID points to the catalog (schema), not the state, which holds the binlog position. I'd have to dig into this a bit to determine if the intended behavior is to get a new

catalogId

when the schema hasn't changed, but my understanding is that this would cause some duplication in the config database at worst.

Hey @Matt Bayley, after your clarification we dug into the logs and found something more concrete. The issue isn't that the state is lost — it's that the state never advances, even though a checkpoint is emitted at the end of every sync. Sync 605992 (10:06 CEST): • Found previous partition offset:

{file=dbabc84.070806, pos=52663308}

• Reads all the way to

dbabc84.070841/33986269

(32K records) •

checkpoint of 1 state message(s)

emitted — sync completes successfully Sync 606240 (11:06 CEST, one hour later): • Found previous partition offset:

{file=dbabc84.070806, pos=52663308}

← exact same offset This repeats for every subsequent hourly sync. The checkpoint is emitted by the source but the state in the Airbyte platform DB is never updated beyond

070806/52663308

. One other thing we noticed:

generateColdStartOffset

is called at the beginning of every sync (not just when there's no prior state). Not sure if that's expected or relevant here. Happy to share the full logs if it helps.

And this isn't an idle database, correct? Sure, send over the logs.

Hi, correct: this isn't an IDLE database. Attached are sanitized logs from three consecutive syncs of our

finance_automation

CDC connection (MySQL → BigQuery, Airbyte 2.1.0,

source-mysql:3.51.5

). The behaviour in a nutshell: the CDC state (binlog offset) is emitted as a checkpoint by the source but never persisted — so every sync restarts from the same stale position. What the logs show: • Sync 605745 (09:06 CEST, June 18) — healthy sync. Reads from

dbabc84.070735/15615386

, advances to

070806

, emits

checkpoint of 1 state message(s)

. • Sync 605992 (10:06 CEST, +1h) — first stuck sync.

getPreviousOffsets

returns

{file=dbabc84.070806, pos=52663308}

— correct, it picked up the offset from 605745. Reads binlog from

070806

to

070841

, emits

checkpoint of 1 state message(s)

. All looks fine. • Sync 606240 (11:06 CEST, +1h) — confirms the bug.

getPreviousOffsets

again returns

{file=dbabc84.070806, pos=52663308}

— the exact same offset as the previous sync, despite 605992 having emitted its checkpoint. The state was never saved. From this point on, every hourly sync reprocesses a growing window of binlog (from

070806

to whatever the current position is), causing exponentially increasing

records_emitted

. Additional observation:

generateColdStartOffset

is called at the very start of every sync (not just cold starts), constructing a synthetic offset pointing to the current binlog tip. We're not sure if this is intentional and whether it could interfere with how the real offset is loaded or committed. Let me know if you need more syncs or the full raw logs.

It's a bit disorienting only seeing a subset of the logs. You could zip up the full ones and DM them to me if you'd like. Another thing you could check yourself is whether the state is getting advanced between syncs. You can access the

state

table in the Postgres instance and check whether the binlog position is advancing there. Also, you could view the state in the UI under Settings -> Connection State, save that somewhere, run a sync, and compare the new state to the old one.