Hello, I need some help with an issue: one of my synchronizations ran for 107 hours with no result. I stopped it manually, but since then I haven’t been able to use this connection at all. Each time I try to run it, I get the following error:

Saved offset is not valid. Please reset the connection, and then increase oplog retention and/or increase sync frequency.

I can’t simply reset the connection, since some of the streams are incremental - resetting would cause significant data loss and would be very difficult to recover. Do you have any advice on how to restart this connection without wiping the existing data?

I'm sorry you're experiencing this issue. Based on the knowledge sources, the error “Saved offset is not valid. Please reset the connection, and then increase oplog retention and/or increase sync frequency” is a known problem with the MongoDB source connector in Airbyte, especially after a sync is interrupted or runs for a long time. This typically happens when the saved resume token (offset) is no longer present in the MongoDB oplog, often due to oplog retention limits or sync frequency being too low. Unfortunately, the official recommendation from Airbyte is to reset the connection, which, as you noted, will cause incremental streams to start from scratch and may lead to data loss or duplication. There is no officially documented way to recover the connection state and resume incremental syncs without a reset. However, some users have reported a workaround by manually editing the connection state to update the resume token. This involves copying the up-to-date resume token from a working state key (if available) and replacing the stale one. For example, if your connection state contains two keys (such as one for the MongoDB URL and another for a server ID), you might be able to copy the valid resume token from the key that is still updating to the one that is stale. This is a manual process and not officially supported, so proceed with caution:

json "shared_state": {   "state": {     "[\"prod\",{\"server_id\":\"prod\"}]": "<old_stale_resume_token>",     "[<mongo_url>]": "<new_resume_token>"   },   "schema_enforced": false }

Replace

<old_stale_resume_token>

with

<new_resume_token>

if the latter is up-to-date and valid. Users have reported that this allows the sync to proceed without a full reset, but it is not guaranteed and may only be a temporary fix. Also, this workaround may not be applicable in all cases, especially if both tokens are stale or missing. You should back up your state before making any manual changes. [See user reports and discussion](https://github.com/airbytehq/airbyte/issues/46569) Other suggestions from the knowledge sources: - Ensure your MongoDB user has

read

permission on the

local

database so Airbyte can correctly read oplog stats. This can help prevent the issue in the future, but does not resolve a currently invalid offset. - Increase your oplog retention and/or sync frequency to reduce the chance of the offset being lost in the future. - If you are using a version of the MongoDB connector prior to 2.0.4, consider upgrading, as there have been fixes related to state handling and resume tokens in newer versions. However, some users still report issues even on 2.0.4, so this is not a guaranteed fix for your current state. Summary: There is no officially supported way to recover from this error without resetting the connection. The only known workaround is to manually update the state with a valid resume token, but this is risky and not guaranteed. If you cannot risk data loss, you may want to reach out to Airbyte support directly for further assistance or wait for a connector update that addresses this issue more robustly. For more details and user experiences, see the related discussions: - [Incremental syncs giving errors - Saved offset is not valid](https://github.com/airbytehq/airbyte/issues/46569) - [Incremental sync error on specific collection - Saved offset is not valid](https://github.com/airbytehq/airbyte/issues/48496) If you need step-by-step guidance on manually editing the state, let me know, but please be aware of the risks involved.

i need step-by-step guidance on manually editing the state @kapa.ai

A message from kapa.ai