@kapa.ai what should I do when I get Saved offset is not valid. Please reset the connection, and then increase oplog retention and/or increase sync frequency to prevent his from happening in the future. See https://docs.airbyte.com/integrations/sources/mongodb-v2#mongodb-oplog-and-change-streams for more details I cannot inxrease the frequency of the sync, it must run once daily, why do I get this, how can I bullet proof this ?

You’re hitting a MongoDB CDC limitation: the resume token Airbyte saved is no longer valid because the corresponding entry has fallen out of the MongoDB oplog before the next sync runs. The connector surfaces this as: > Saved offset is not valid. Please reset the connection, and then increase oplog retention and/or increase sync frequency… The official guidance for this connector is: • Ensure the oplog holds at least 24 hours of changes; a week’s worth is recommended. [MongoDB v2 troubleshooting] • If you see “invalid resume token” / “Saved offset is not valid”, you must: a. Increase oplog retention period and/or size, and b. Increase Airbyte sync frequency (so at least one successful sync advances the cursor before the oplog recycles). [MongoDB v2 troubleshooting; Issue 62833] Because you’re constrained to once-daily syncs, the only lever you really have (per the docs and issues) is: 1. Increase oplog retention well beyond 24 hours ◦ Configure your MongoDB cluster so the oplog keeps multiple days of history (the docs explicitly recommend about a week). [MongoDB v2 troubleshooting] ◦ In practice, this needs to be long enough that: ▪︎ A full sync (if large) can complete, and ▪︎ At least one incremental sync runs and reads some changes before the earliest oplog entries are recycled. [MongoDB v2 troubleshooting; Issue 62833] 2. Reset the connection when this happens ◦ The error message and issues indicate that once the offset is invalid, you must reset the connection / re-import the affected collections to avoid missing data. [Issue 62833; Issue 70222] 3. Make sure at least one sync within the retention window actually reads changes ◦ A contributor notes that if no data movement occurs and the LSN (cursor) doesn’t advance before the oplog recycles, you can still lose the position even if retention is 24h. [Issue 62833] ◦ With daily syncs, this means your oplog window must comfortably exceed 24h so that “yesterday’s” cursor is still valid when today’s sync starts. To “bullet‑proof” under a strict once‑per‑day schedule, the only measures explicitly supported by the docs and issues are: • Substantially increase oplog size/retention (aim for about a week or more, depending on your write volume and initial load duration). [MongoDB v2 troubleshooting] • Accept that when the error occurs you must reset/reload the connection to avoid data loss. [Issue 62833] The knowledge sources do not provide any additional “bullet‑proof” mechanism beyond these oplog and frequency recommendations. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai in my azure storage account, what wil happen if i resfresh data from a connection with mongodb as source and storage account as destination, I've had incremental syncs happening for 3 days until now

In Airbyte, a Refresh is a special operation that re-reads data from the source and writes it again to the destination, with two main variants: 1. Refresh and remove records – old data is removed if it no longer appears in the source. 2. Refresh and retain records – old data is kept and new data is merged in, so storage grows. [Refresh & retain] For your case: • Source: MongoDB (with 3 days of incremental sync history) • Destination: Azure Blob Storage (supports Full Refresh and Incremental | Append, but not Incremental | Append + Deduped). [Azure Blob Storage] What happens on Azure Blob Storage depends on the sync mode + refresh type you choose: • With Full Refresh | Overwrite (or a Refresh that removes records), each sync overwrites the destination data with a fresh copy of the source. Old blobs corresponding to that stream’s current namespace/prefix are effectively replaced by the new load (implementation details for blob deletion vs. replacement are not described in the docs, so I can’t be more specific). [Full refresh tutorial; Refresh FAQ] • With Full Refresh | Append (or a Refresh that retains records), each refresh appends a full copy of the source data as new blobs, keeping all previous data. This increases storage usage over time. [Full refresh tutorial; Refresh & retain] Your previous 3 days of incremental syncs: • Are just earlier writes into Azure Blob Storage (each stream written as separate blobs under the container). [Azure Blob Storage] • A subsequent Refresh does not “merge” with those incremental blobs at the file level; instead, it behaves like a full refresh according to the chosen mode (overwrite vs append). The docs do not describe any automatic cleanup of older incremental blobs beyond what overwrite semantics imply, so exact blob-level behavior is not fully specified. So, practically: • If you run a Refresh that retains records / Full Refresh | Append, you will end up with: ◦ Existing incremental blobs from the last 3 days plus ◦ New full-refresh blobs, increasing total storage. • If you run a Refresh that removes records / Full Refresh | Overwrite, the destination will reflect only the latest full snapshot of MongoDB for that connection/stream; older data for that stream is considered removed/overwritten in the destination. [Refresh FAQ] The knowledge sources do not provide a more detailed, blob-by-blob description of how older incremental files are handled during overwrite in Azure Blob Storage. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai New full-refresh blobs, that will include all the new data and the old data as well, so basically duplicated ?

Yes—when you use Full Refresh | Append, each sync writes the entire current dataset again and does not touch existing data. That means: • The new blobs will contain all records currently in MongoDB (both “old” and “new” from the source’s perspective). • Previous full-refresh or incremental blobs remain in Azure Storage. • So, from a storage perspective, you will indeed have multiple copies of the same records across different blobs (logical duplication). [Full refresh append; Full refresh tutorial] This is exactly how Full Refresh | Append is defined: “add new data to the same table, do not touch existing data,” and “every sync will create duplicates of already existing data.” [Full refresh append] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai do I need to increase teh oplog size, or the op,log window ?

You need to ensure the oplog window (i.e., how many hours/days of history the oplog retains) is long enough for your sync pattern. You typically achieve that by increasing the oplog size, which in turn increases the window. The docs and issues talk about it in terms of retention / window, not just raw size: • Airbyte recommends configuring MongoDB so the oplog holds at least 24 hours of changes, and ideally about a week’s worth. [MongoDB v2 troubleshooting] • If you see “invalid resume token” / “Saved offset is not valid”, the guidance is to: a. Increase the Oplog retention period / size, and b. Increase Airbyte sync frequency, so at least one successful sync advances the cursor before the oplog recycles. [MongoDB v2 troubleshooting; Issue 62833] So in practice: • What you care about is the time window (how many hours of operations are kept). • To enlarge that window, you increase the oplog size in MongoDB until the reported

log length start to end

(the oplog window) comfortably exceeds your sync interval (for daily syncs, usually several days). [MongoDB v2 troubleshooting] The knowledge sources do not provide more detailed tuning formulas beyond “at least 24 hours, preferably a week,” but they consistently frame the requirement as having a sufficiently long oplog retention window, achieved by increasing oplog size. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai My airbyte sync to azure stoarage account writes files to maxx 200 mb before moving to another csv file . And somethins it creats 2 different files with 10 mb and 7 mb each . Why 2 files? when the setting sis How large each blob should be, in megabytes. Example: 5​00. After a blob exceeds this size, the connector will start writing to a new blob, and increment the part number. set to 500

The knowledge sources confirm how the Azure Blob Storage destination is supposed to behave, but they don’t fully explain the exact file sizes you’re seeing. What we know: • The connector has a setting: “Azure Blob Storage Target Blob Size (MB)” – “How large each blob should be, in megabytes. After a blob exceeds this size, the connector will start writing to a new blob, and increment the part number.” [Azure dest config] • In the Terraform schema this is called

azure_blob_storage_spill_size

/ “file split size”: “The amount of megabytes after which the connector should spill the records in a new blob object. Make sure to configure size greater than individual records. Enter 0 if not applicable. Default: 500.” [Azure TF schema; Azure migration guide] So, conceptually: • Once a blob exceeds the configured size (500 MB in your case), the connector starts a new blob/part. • The docs do not guarantee that: ◦ Every blob will be exactly that size, or ◦ That there will only ever be one blob per sync/stream. Reasons you might see multiple smaller files (e.g., 10 MB and 7 MB) even with a 500 MB target, based on the docs and related behavior in other bucket destinations: • The split size is a maximum threshold, not a minimum; smaller blobs can be produced if the sync ends before the threshold is reached (e.g., low volume, end of stream). [Azure dest config] • The migration guide notes that the “file split size” option now applies to CSV and that the connector writes block blobs and treats each blob as complete once written. It doesn’t describe any guarantee about minimum size or single-file-per-sync behavior. [Azure migration guide] The knowledge sources do not explain: • Why, in a specific run, you get two small blobs (10 MB and 7 MB) instead of one ~17 MB blob. • Whether internal batching, parallelism, or per-stream partitioning is causing that exact pattern. Given the available docs, all we can say confidently is: • The 500 MB setting is a cap after which a new blob is started, not a promise that blobs will be close to 500 MB. • Multiple smaller blobs per sync/stream are expected and supported behavior; the docs don’t provide a way to force a single file or tightly control exact blob sizes. [Azure dest config; Azure migration guide] If you need stricter guarantees (e.g., exactly one file or minimum size), the current documentation does not describe any configuration to achieve that. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.