https://linen.dev logo
Join Slack
Powered by
mongodb-v2-source - resume token invalid if no new...
# replication-ask-ai
t
mongodb-v2-source - resume token invalid if no new data comes into the source database before the oplog retention is no longer valid
k
This behavior is expected given how the MongoDB v2 source uses change streams and the oplog. MongoDB change streams are backed by the replica set oplog, which has a finite retention window. If: - your Airbyte sync runs less frequently than the oplog retention window, or - there are no changes (no CDC events) for a period longer than the oplog window, then the resume token stored in Airbyte’s state can point to an oplog position that has already been purged. When the connector tries to resume from that token, MongoDB returns errors like 
ChangeStreamHistoryLost
/ “resume token was not found”, which Airbyte surfaces as 
Saved offset is not valid
/ “invalid resume token”. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations); [Issue 62833](https://github.com/airbytehq/airbyte/issues/62833)] From the docs and issues, the recommended mitigations are: 1. Increase oplog retention / size so it covers at least 24 hours of changes; a week is recommended. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations)] 2. Increase sync frequency so that at least one successful sync that reads changes occurs within each oplog window, ensuring the cursor/LSN advances before older entries are recycled. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations); [Issue 62833](https://github.com/airbytehq/airbyte/issues/62833)] 3. If you already hit the error, you typically need to reset the connection and re‑import the collection to guarantee no changes were missed. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations)] The knowledge sources do not describe any connector-side workaround beyond these configuration changes; they explicitly tie “invalid resume token” / “Saved offset is not valid” to oplog history loss and insufficient sync frequency.
t
@kapa.ai we're doing frequent syncs, but there is not any new data in the source database to
k
A message from kapa.ai
You’ve hit an important edge case that’s only partially covered in the current materials. From the docs and issues, we know: - The MongoDB v2 source relies on the oplog; if the resume point is no longer in the oplog, MongoDB returns 
ChangeStreamHistoryLost
, which Airbyte surfaces as 
Saved offset is not valid
/ “invalid resume token”. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations); [Issue 62833](https://github.com/airbytehq/airbyte/issues/62833)] - Contributors explicitly say that within the oplog window you need **at least one successful sync that reads changes and advances the LSN**; otherwise, “you can lose the position and the following sync will fail.” [[Issue 62833](https://github.com/airbytehq/airbyte/issues/62833)] This implies: - If you are syncing frequently but no new data is written for longer than the oplog retention, the stored resume token can still become invalid, because the underlying oplog entry it refers to is eventually purged. - The connector currently expects that at least some CDC activity happens within each oplog window; it does not have a documented special handling for “completely idle” periods. The docs do not describe a connector-side fix for this “no new data” scenario beyond the same recommendations: 1. Ensure the oplog retention window is long enough to cover the longest expected period with no writes (not just the sync interval). [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations)] 2. If the error already occurred, reset the connection and re‑import to avoid missing changes. [[MongoDB troubleshooting](https://docs.airbyte.com/integrations/sources/mongodb-v2/mongodb-v2-troubleshooting#connector-limitations)] If you can share: - Your oplog retention (hours), - Typical “no‑write” durations, - Connector version (e.g. 2.0.4), I can map your situation more precisely to the patterns described in the open/closed issues.
2 Views
Open in Slack
PreviousNext
Powered by