<@U04S8PLJWMT> Hi there, I’m running into an issue...
# ask-ai
a
@kapa.ai Hi there, I’m running into an issue with a Postgres CDC → Snowflake connection. Since Monday, all syncs complete successfully, but 0 records are replicated across all streams. After checking the connection state, I noticed something strange: • The shared CDC state (Debezium) contains a valid LSN and is advancing normally. • But all streamStates only contain cursor_field: [] and no per-stream state at all. • Logs show for every stream: • Found cursor field. Original cursor field: null. New cursor field: _ab_cdc_lsn. Resetting cursor value. • Debezium logs also show: • Received offset commit request… but ignoring it. LSN flushing is not allowed yet. The replication slot on Postgres is healthy and active,. It looks like Airbyte is stuck in an inconsistent snapshot/CDC state. The result is that no changes are produced during syncs. Could you help me understand how to fix this state inconsistency, and whether a full “Reset data + Reset state” is required? Is there any way to recover the connection without triggering a full snapshot? Thanks!
k
Based on the information in your query and the available knowledge sources, it appears you are encountering a known issue where the Postgres CDC connector gets into an inconsistent state, resulting in successful syncs with 0 records and no per-stream state being tracked. This is often related to issues with the Log Sequence Number (LSN) management and state handling between Airbyte and Debezium. Key Points from Knowledge Sources: - There are multiple reports of similar issues where the CDC state advances, but no records are replicated, and the per-stream state is missing or not updated. This can happen if the connector commits the wrong LSN or if the state is not properly updated after the initial snapshot, especially in low-traffic databases or after certain upgrades [[Github Issue 49803](https://github.com/airbytehq/airbyte/issues/49803)]. - The logs you mention (“Received offset commit request… but ignoring it. LSN flushing is not allowed yet.”) are consistent with situations where Debezium is not allowed to flush the LSN, often due to a mismatch between the saved state and the confirmed flush LSN in the replication slot [[Github Issue 55846](https://github.com/airbytehq/airbyte/issues/55846)]. - In some cases, the only way to recover is to reset the connection state, which may require a full resync. However, users have reported that even after manually updating the state to match the confirmed_flush_lsn, the issue can recur on the next sync [[Github Issue 55846](https://github.com/airbytehq/airbyte/issues/55846)]. - There are also suggestions to increase the “Initial Waiting Time in Seconds (Advanced)” parameter and ensure the WAL size is sufficient, but these are more relevant to timeouts and WAL retention issues, not state inconsistencies [[Github Issue 50435](https://github.com/airbytehq/airbyte/issues/50435)]. Recommended Actions: 1. Reset Data and State: Unfortunately, the most reliable way to recover from this state inconsistency is to perform a full “Reset data + Reset state” on the connection. This will force Airbyte to take a new initial snapshot and re-align the CDC state. There is no documented way to recover the connection without a full snapshot if the per-stream state is missing or corrupted [[Github Issue 55846](https://github.com/airbytehq/airbyte/issues/55846)]. 2. Upgrade Connectors: Make sure you are using the latest versions of both the Postgres source and Snowflake destination connectors, as several bugs related to CDC state management and LSN handling have been fixed in recent releases [[Forum Post](https://discuss.airbyte.io/t/4373)]. 3. Monitor for Recurrence: If the issue recurs, consider enabling heartbeat events or adding a high-transaction table to the publication to ensure regular WAL advancement, especially in low-traffic environments [[Postgres Troubleshooting](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#advanced-wal-disk-consumption-and-heartbeat-action-query)]. 4. Check for Known Bugs: If you continue to experience issues after resetting and upgrading, review the open and recently closed issues on the Airbyte GitHub repository for any new patches or workarounds. Summary: A full reset is likely required to resolve the state inconsistency. There is no supported way to recover the connection and resume CDC without a new snapshot if the per-stream state is missing or not advancing. Upgrading connectors and monitoring for future issues is also recommended. If you need more targeted help, please provide your connector versions and any recent changes to your Airbyte or Postgres setup.
a
@kapa.ai should i refresh the data, clear the data or delete and recreate the connection ?
k
A message from kapa.ai
Given your situation with a Postgres CDC → Snowflake connection where syncs complete but 0 records are replicated, here’s how the Airbyte actions differ and what is recommended: ### Refresh Data - A Refresh Sync erases the current cursor for incremental streams and re-imports data from the beginning, but does not delete existing data until the new data is ready. It is useful if you want to re-attempt syncing or recover from a bad state, but it does not clear out the destination tables entirely. It does not create data downtime, as the previous data remains available until the refresh completes. However, if the state is corrupted or missing (as in your case), a refresh may not resolve the underlying issue [[Refreshing your data](https://docs.airbyte.com/platform/operator-guides/refreshes)]. ### Clear Data - Clear Data (previously called "Reset") erases all data that Airbyte has created in your destination for the selected streams or the entire connection. This is a destructive operation: all records are deleted from the destination tables, and you will need to run a sync to reload data. This is recommended if you want to start fresh and ensure no corrupted or partial data remains. Note: this causes data downtime, as the tables will be empty until the next sync completes [[Clearing your data](https://docs.airbyte.com/platform/operator-guides/clear)]. ### Delete and Recreate the Connection - Deleting and recreating the connection is the most drastic step. This will remove all configuration, state, and history for the connection. You would need to set up everything from scratch. This is only recommended if other recovery options fail or if you suspect the connection configuration itself is corrupted. --- ### What Should You Do? Based on the knowledge sources, if your connection is stuck in an inconsistent state (e.g., missing per-stream state, no records replicated), the recommended approach is to Clear Data for the affected streams or the entire connection. This will remove all previously synced data and allow you to start with a clean slate. After clearing, run a new sync to backfill the data. If clearing does not resolve the issue, then consider deleting and recreating the connection as a last resort. Summary: - Try "Clear Data" first to reset the destination and state. - If issues persist after clearing and resyncing, then delete and recreate the connection. > There is no way to recover from a clear sync, so please be certain that you wish to erase all the data in your destination. > In order to backfill all historical data, a sync should be initiated after your clear succeeds [[Clearing your data](https://docs.airbyte.com/platform/operator-guides/clear)]. If you only want to re-import data without deleting it first, use "Refresh." But for state corruption, "Clear Data" is the more robust option. If you need step-by-step instructions for clearing data, let me know!