airbyte still trying cdc even using full orverwrit...
# replication-ask-ai
r
airbyte still trying cdc even using full orverwrite, error:
Copy code
Failure in source: Saved offset no longer present on the server. Please reset the connection, and then increase binlog retention and/or increase sync frequency.
k
The knowledge sources don't explicitly cover the scenario of CDC errors occurring when using Full Overwrite sync mode. However, based on what's available, here's what's likely happening and how to address it: What's Happening The error
Saved offset no longer present on the server
is a CDC state validation error that occurs at the source connector level, before Airbyte even considers the destination sync mode. Even if your stream is set to Full Overwrite on the destination side, if the source connection's update method is still set to CDC, Airbyte will attempt to validate the saved CDC offset (LSN) on startup — and fail if that offset has been cleaned up by SQL Server's CDC retention job. [GitHub Issue] The sync mode (Full Overwrite) controls how data is written to the destination, not how it's read from the source. The source read method (CDC vs. Full Refresh) is a separate setting. How to Fix It 1. Change the source update method to Full Refresh instead of CDC. In your connection's stream configuration, set the sync mode to
Full Refresh
on the source side. This bypasses CDC entirely and avoids LSN validation. 2. Reset the connection — if you must keep CDC, you need to reset the connection state so Airbyte picks up a fresh LSN. Go to your connection settings and trigger a connection reset. 3. Increase SQL Server CDC retention — configure SQL Server to retain CDC data longer so the saved LSN doesn't expire between syncs. 4. Increase sync frequency — sync more often (e.g., every 5 minutes) so the LSN is consumed before CDC cleanup removes it. [GitHub Issue] 5. Upgrade source-mssql — recent fixes (versions `4.4.3`+,
4.4.8
) addressed false-negative LSN validation failures and idle CDC state refresh issues. [PR #77669] [PR #78299] Summary The key point is: Full Overwrite is a destination write mode, not a source read mode. To stop CDC from being used, you need to explicitly change the source sync mode to
Full Refresh
in your stream's configuration. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai that's my connection state:
k
Sorry @Rafael Felipe, I encountered an error processing the file(s) you uploaded: File type is not supported
r
@kapa.ai that's my connection state: ``````
k
Sorry @Rafael Felipe, I encountered an error processing the file(s) you uploaded: File type is not supported
r
@kapa.ai how I reset connection state?
k
Based on the knowledge sources, here are the ways to reset/clear your connection state: Option 1: Clear Data (Full Connection) 1. Navigate to your connection in the Airbyte UI 2. Go to the Settings tab 3. Click Clear data 4. Confirm the selection This removes all previously synced data and resets the state for all streams. [Clear Data Docs] Option 2: Clear a Single Stream 1. Navigate to your connection's Status page 2. Click the three grey dots next to the stream 3. Select Clear data 4. Re-run the sync afterward to reload data for that stream [Clear Data Docs] Option 3: Refresh Sync 1. Trigger a Refresh and Remove Records or Refresh and Retain Records sync from the connection UI 2. This resets the cursor/state without causing data downtime (previous data remains queryable until the refresh completes) [Refresh Docs] Option 4: Manually Edit Connection State 1. Go to your connection in the Airbyte UI 2. Click the Settings tab 3. Click the Advanced dropdown 4. Click in the Connection State field to edit it directly 5. Click Update state to confirm > ⚠️ Manual state edits should be handled with extreme care, as incorrect changes may break your syncs. [Connection State Docs] For your CDC offset issue specifically, Option 1 or Option 3 (Remove Records) are the most straightforward approaches. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai Update state is grayout
k
The knowledge sources do not contain enough information to explain specifically why the Update State button would be grayed out or how to resolve it. A few possible reasons based on general context: • You may not have the required permissions (e.g., workspace admin role) to edit connection state. • The connection may be actively syncing — try waiting for the sync to finish before editing. • There could be a browser caching issue — one GitHub issue noted that clearing browser application data resolved a similar grayed-out button problem. [GitHub Issue] Recommended Next Steps 1. Clear your browser cache completely and reload the page. 2. Ensure you have admin-level permissions on the workspace. 3. Make sure no sync is currently running on the connection. 4. If you're on Airbyte Cloud, consider opening a support ticket at support.airbyte.com or reaching out via the Airbyte Community Slack for further assistance. I'm sorry I can't be more specific — the knowledge sources don't explicitly document this behavior. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
r
@kapa.ai which field from connection state I should update to cleanup cdc config?
k
The knowledge sources do not contain enough specific information to give a definitive, safe answer about exactly which field(s) to modify in the connection state JSON to clean up a CDC state. However, based on what the knowledge sources do show, the CDC state structure for common connectors looks like this: For MSSQL/Postgres (example):
Copy code
{
  "shared_state": {
    "state": {
      "[\"db_name\",{\"server\":\"db_name\"}]": "{\"transaction_id\":null,\"lsn\":54076598925816,...}"
    }
  },
  "streamStates": [
    {
      "streamDescriptor": {
        "name": "your_table",
        "namespace": "your_schema"
      },
      "streamState": { ... }
    }
  ]
}
The
shared_state
contains the CDC offset (LSN) that is causing the error. [GitHub Issue] ⚠️ Important Warning > Updates to connection state should be handled with extreme care. Updates may break your syncs, requiring a full historical sync to fix. Make changes only as directed by the Airbyte team. [Connection State Docs] My Recommendation Rather than manually editing the state JSON (which risks breaking things further), I'd strongly suggest: 1. Reaching out to Airbyte Support at support.airbyte.com for guided assistance. 2. Joining the Airbyte Community Slack where team members can advise on the exact field to clear for your specific connector and version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
m
Hey @Rafael Felipe, are you still having trouble? Maybe I can help.
r
I did start a new connection. Sometimes airbyte try to resume binlog read from some unknown date point, even if we refresh data...so I created another connection
thanks
m
yeah, when a connection is created in CDC mode, it will validate the binlog availability, even if all the streams are set to full refresh
if you don't want to deal with that, you can create a user-cursor or xmin connection instead
r
yes, you are right. Appreciate your help.
👍 1