@kapa.ai I have a connection between 2 mongodb databases. I have about 4 million documents to load. Everything goes well but at the end of the process, probably when the documents are needed to be written from airbyte_tmp collection to airbyte_raw collections, the destination spike seriously in memory. I have increased the memory limit to a very high amount, but that does not solve my issue the memory spikes heavily., I want to avoid that, how can I avoid that ?

This is a known bug in the destination-mongodb connector. The root cause is that the

copyTable

function loads all documents into memory at once (without batching) when copying from the temporary collection to the permanent collection. With 4 million documents, this will cause a massive memory spike regardless of how much memory you allocate. [destination-mongodb OOM issue] What's happening under the hood The current code fetches all documents from the temp collection and inserts them into the permanent collection in one go, exhausting the JVM heap. The issue reporter proposed a fix using batched inserts: [destination-mongodb OOM issue]

private static void copyTable(final MongoDatabase mongoDatabase, final String collectionName, final String tmpCollectionName) {
    final var tempCollection = mongoDatabase.getOrCreateNewCollection(tmpCollectionName);
    final var collection = mongoDatabase.getOrCreateNewCollection(collectionName);

    final int BATCH_SIZE = 1000;
    final List<Document> batch = new ArrayList();

    try (final MongoCursor<Document> cursor = tempCollection.find().projection(excludeId()).iterator()) {
        while (cursor.hasNext()) {
            batch.add(cursor.next());

            if (batch.size() == BATCH_SIZE) {
                collection.insertMany(new ArrayList(batch));
                batch.clear();
            }
        }

        if (!batch.isEmpty()) {
            collection.insertMany(batch);
        }
    }
}

What you can do right now Unfortunately, this is an open bug with no confirmed fix released yet. The workarounds available are limited: 1. Reduce batch size: Some users tried reducing the destination batch size (e.g., from 10,000 to 1,000), but reported it did not resolve the OOM issue during the copy phase. [destination-mongodb OOM issue] 2. Split your sync into smaller connections: You could create multiple connections targeting subsets of your data (e.g., by collection), which reduces the volume processed per sync job. [Source Postgres memory discussion] 3. Contribute the fix: The issue is open and the reporter has indicated willingness to contribute. If you have Java development experience, you could submit a pull request implementing the batched

copyTable

logic shown above. [destination-mongodb OOM issue] In summary, simply increasing memory is not a solution because the problem is architectural — the connector needs to be patched to use batched inserts during the copy phase. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai On another note A connection from Mongodb source to a Mongodb destination has this statistics 2.51 GB OR 2,851,579 records took to sync 3h 10m 34s Is there a way to make Airbyte OSS load this amount of documents, in a shorter time ?

That's a reasonable concern — ~2.5 GB / ~2.85M records in ~3 hours is quite slow. Here's what the knowledge sources say about improving sync performance in Airbyte OSS: Options to Improve Sync Speed 1. Upgrade to Airbyte Self-Managed v1.8+ Airbyte has introduced a "Faster sync speed" initiative that delivers 4–10x speed improvements for supported connectors. Benchmarks show syncing a 170 GB MySQL-to-S3 dataset went from 1h 35m down to 22 minutes on C4 nodes. [faster sync speed] To benefit from this: • Upgrade to Self-Managed v1.8 or later • Use connector versions that support faster syncs • Note: the exact gains depend on your infrastructure and whether MongoDB connectors are included in the supported set [faster sync speed] 2. Use High-Performance CPU Nodes The benchmarks show that CPU type is now directly correlated with sync speed. High-performance CPUs (e.g., C4 on GCP) yielded the best results, but even medium-performance CPUs (N2D) delivered significant improvements. [faster sync speed] 3. Allocate More Resources to Connector Pods You can tune the CPU and memory allocated to sync job containers using environment variables: [sync performance tips]

JOB_MAIN_CONTAINER_CPU_REQUEST=
JOB_MAIN_CONTAINER_CPU_LIMIT=
JOB_MAIN_CONTAINER_MEMORY_REQUEST=
JOB_MAIN_CONTAINER_MEMORY_LIMIT=

4. Split Into Multiple Connections If your data spans multiple collections, splitting them into separate connections allows parallel syncing, which can reduce total wall-clock time. [sync performance tips] 5. Ensure Sufficient Kubernetes Resources Make sure your Kubernetes cluster can schedule enough pods. A sync job uses two workers (one for source, one for destination), so ensure your cluster has headroom. [scaling Airbyte] ------------------------- Important caveat: The knowledge sources do not explicitly confirm whether the MongoDB-to-MongoDB connector pair is among those covered by the faster sync speed initiative. You should verify this in the Airbyte documentation or changelog for your specific connector versions. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.