@kapa.ai on open source airbyte my mongodb connection using CDC seems to just keep trying to carry on loading new records as they come in instead of exiting, until it hits a java heap space error

This is a known issue with the MongoDB CDC connector in self-hosted Airbyte. Based on the knowledge sources, there are actually a couple of related but distinct problems that could explain what you're seeing: 1. CDC Sync Not Stopping (Keeps Reading New Records) The

source-mongodb-v2

connector uses a target position mechanism based on a resume token timestamp recorded at the start of the sync. The sync should stop when the event timestamp exceeds that target timestamp. If new records keep arriving, the connector may keep processing them instead of stopping at the originally recorded position. [MongoDbCdcTargetPosition] 2. Java Heap Space OOM on the Destination Side There is an open bug in

destination-mongodb

v0.2.0 where, at the end of a sync, the connector copies all records from a temporary collection to the permanent one by loading everything into memory at once — with no batching. On large datasets, this causes

java.lang.OutOfMemoryError: Java heap space

. [GitHub Issue] The relevant log pattern looks like:

Migration finished with no explicit errors. Copying data from tmp tables to permanent
Terminating due to java.lang.OutOfMemoryError: Java heap space

A proposed fix (not yet merged as of the issue) is to batch the copy in chunks of ~1000 documents:

final int BATCH_SIZE = 1000;
final List<Document> batch = new ArrayList<>();

try (final MongoCursor<Document> cursor = tempCollection.find().projection(excludeId()).iterator()) {
    while (cursor.hasNext()) {
        batch.add(cursor.next());
        if (batch.size() == BATCH_SIZE) {
            collection.insertMany(new ArrayList<>(batch));
            batch.clear();
        }
    }
    if (!batch.isEmpty()) {
        collection.insertMany(batch);
    }
}

3. GlobalMemoryManager Negative Accounting (Async Buffer Framework) There is also a separate open bug where

GlobalMemoryManager.free()

can drive

currentMemoryBytes

negative, disabling all backpressure and causing unbounded buffering until OOM. This affects any destination using the async buffer framework. [GitHub Issue] What You Can Try • Reduce sync volume: If using Full Refresh, try limiting the data volume per sync. • Increase JVM heap: Set

JAVA_OPTS: "-XX:MaxRAMPercentage=80.0"

and ensure the connector pod has sufficient memory allocated. • Watch for the fix: Issue #48851 is open and the community has proposed a batching fix — you may want to upvote or contribute to it. • Consider switching destination: Users have reported that syncing to Postgres instead of MongoDB avoids the OOM issue for large streams. [GitHub Issue] These are open bugs with no confirmed fix merged yet, so the issue may persist on the current connector versions. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai my deployment is on v1.2.1

I'm sorry, but the knowledge sources don't contain enough information to give you specific guidance for Airbyte v1.2.1 in relation to this MongoDB CDC OOM issue. What I can say based on the knowledge sources is: • The

destination-mongodb

OOM bug (issue #48851) was reported against connector version 0.2.0 and remains open with no confirmed fix merged, so it is likely still present in your deployment regardless of the Airbyte platform version. • The

GlobalMemoryManager

negative accounting bug (#74897) was reported against Airbyte OSS v2.0.1, so I cannot confirm whether it applies to v1.2.1. I'd recommend: 1. Checking the exact version of the

destination-mongodb

connector you are running (visible in the connection settings), as the platform version and connector version are independent. 2. Opening a support request or GitHub issue referencing #48851 with your specific platform version to get a more targeted response from the Airbyte team. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai can i switch my source mongodb connection to use a cursor field for incremental loading instead of CDC?

Based on the knowledge sources, no — the

source-mongodb-v2

connector does not support cursor-based incremental sync as an alternative to CDC. The documentation states: > "The MongoDB source utilizes change data capture (CDC) as a reliable way to keep your data up to date." The only two replication methods available are: [MongoDB Replication Methods] 1. CDC (Incremental) — uses MongoDB change streams 2. Full Refresh — reads the entire collection repeatedly There is no cursor-based (

updatedAt

or similar) incremental option in

source-mongodb-v2

. This is actually something that has been raised by the community as a gap — the GitHub discussion you can see references a request to "re-introduce cursor-based (

updatedAt

) incremental for mongodb-v2", suggesting it existed in an older version but was removed. [GitHub Discussion] Your Options Given your CDC OOM issue, the practical alternatives are: • Full Refresh — avoids CDC entirely but re-reads all data every sync. Works best for smaller collections. • Tune CDC — adjust your Oplog size and sync frequency to reduce the volume of change stream events processed per sync. [MongoDB Troubleshooting] • Switch Update Capture Mode to Lookup — if you're on Post Image mode, switching to Lookup can reduce change stream event sizes and may help with memory pressure. [Update Capture Mode] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I want to manually change the resume token stored in the stream state config of my mongodb source connection. I have the token value, what else do i need to update to make it work?

Based on the knowledge sources, the CDC state stored for MongoDB has a specific structure. To manually update the resume token, you need to update three fields together, not just the token value itself. State Structure The shared CDC state looks like this:

{
  "state": {
    "[\"<normalized_connection_string>\",{\"server_id\":\"<normalized_connection_string>\"}]": "{\"sec\":<timestamp_seconds>,\"ord\":<increment>,\"resume_token\":\"<your_token>\"}"
  },
  "schema_enforced": true
}

[MongoDbDebeziumStateUtil] What You Need to Update The inner value (a JSON-encoded string) contains three fields that must all be consistent with your resume token: The

sec

and

ord

values are derived directly from the

BsonTimestamp

embedded in the resume token itself — they are not arbitrary values. If they don't match the token, the connector may behave unexpectedly. [MongoDbDebeziumStateUtil] Important: The State Key Format The key of the state object also matters. It must match the normalized form of your MongoDB connection string. Based on the known issue with state corruption, if this key doesn't match what the connector expects, it will treat the state as needing migration or as invalid. [GitHub Issue] For example, a connection string like

mongodb+srv://server.cluster.mongodb.net/

gets normalized to

mongodb_srv___server.cluster.mongodb.net_

as the

server_id

. Recommendation Rather than editing the state manually, be aware that: 1. If the `sec`/`ord` values don't match the token, the connector may log a warning but will still attempt to use the

resume_token

field directly for resumption. 2. If the key format is wrong, the connector will detect it as an old/corrupted format and attempt automatic migration — which may or may not recover correctly. [MongoDbCdcInitializer] The knowledge sources don't explicitly document a supported manual state editing procedure, so proceed with caution and consider taking a backup of the existing state before making changes. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

@kapa.ai what does this message mean?

BufferingChangeStreamCursor$EventFetcher(enqueue):285 Unable to acquire buffer lock, buffer queue is likely full

The knowledge sources do not contain enough information to give a precise explanation of that specific log message from

BufferingChangeStreamCursor$EventFetcher.enqueue

at line 285. However, based on what the knowledge sources do describe about how

BufferingChangeStreamCursor

works, I can offer some context: What It Likely Means The

BufferingChangeStreamCursor

uses an internal buffer queue to hold change stream events fetched from MongoDB before they are consumed by the Debezium engine. The

enqueue

method attempts to acquire a lock to place a new event into that buffer. This message suggests: • The fetcher thread tried to add a new change event to the internal buffer queue • It could not acquire the lock — most likely because the buffer is full and the consumer (Debezium engine) is not draining it fast enough This is essentially backpressure from a full internal buffer — MongoDB is producing events faster than the connector can process and forward them. [GitHub Issue] Why It Might Be Happening Based on the knowledge sources, common causes include: • Large change events (~833KB per event with

change_streams_update_full_with_pre_image

) filling the queue quickly [CDK PR] • High write volume on the source collection overwhelming the connector's processing capacity [GitHub Discussion] What You Can Do • If you're using

change_streams_update_full_with_pre_image

as your update capture mode, consider switching to Lookup mode to reduce event sizes [MongoDB Troubleshooting] • Reduce the CDC Queue Size below the default of 10,000 if memory is constrained [CDC Best Practices] For a definitive explanation of that exact log message, I'd recommend checking the Debezium source code for

BufferingChangeStreamCursor

directly, as the knowledge sources don't include that file. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.