@kapa.ai any plans on this being fixed? github.com/airbytehq/airbyte/issues/48851

Thinking completed Based on the provided knowledge sources, there is no information or official timeline regarding plans to fix this issue [destination-mongodb issue]. The issue remains in an open state. While a user requested an update and asked if there is any possibility of a fix, no official resolution, assigned milestone, or planned release has been documented by the Airbyte team in the records [destination-mongodb issue]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai that's great so how are users supposed to make this work if the sync keeps crashing ebcause of large amount of data/

Thinking completed The knowledge sources do not provide an officially verified solution or workaround for this specific bug in the MongoDB destination connector. The underlying problem documented in the issue is that during the final migration step,

MongodbRecordConsumer

loads all documents from the temporary collection into an in-memory list before copying them to the permanent collection without batching, which exhausts the Java heap space when dealing with large volumes of data [MongodbRecordConsumer.java; destination-mongodb issue]. Based on the provided documentation and community discussions, here are the approaches users have explored when encountering memory crashes of this type: 1. Increase Container Memory Limits and JVM Heap You can allocate more memory to the connector containers to accommodate the large in-memory list: • Connection-level resource settings (Kubernetes / Database): Users update the

resource_requirements

column directly on the

connection

table in the Airbyte database [Java heapspace issue; destination-s3-data-lake issue]:UPDATE connection SET resource_requirements = '{"memory_limit": "20Gi", "memory_request": "2Gi"}' WHERE id = 'your-connection-id'; • Environment variables: In Docker Compose

.env

or Helm

values.yaml

, increase job container limits [Forum Post: How to specify CPU,MEMORY limit in .env; Java heapspace issue]:JOB_MAIN_CONTAINER_MEMORY_REQUEST=4g JOB_MAIN_CONTAINER_MEMORY_LIMIT=8g • JVM Options: In Kubernetes environments, configure

JAVA_OPTS

to allow the JVM to utilize the container's RAM [Java heapspace issue]JAVA OPTS "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=80.0 -XX:+UseG1GC"_(Note: As reported in related issues, simply increasing memory may delay the crash rather than prevent it if data volumes significantly exceed available RAM [Forum Post: Destination MongoDB: unexpected high memory usage; destination-s3-data-lake issue].)_ 2. Community Proposed Fix (Custom Connector Build) In the open issue, a community contributor proposed updating

copyTable

in

MongodbRecordConsumer.java

to iterate over a cursor and write in batches (such as batches of 1,000 documents) instead of attempting to copy everything at once [destination-mongodb issue]:

private static void copyTable(final MongoDatabase mongoDatabase, final String collectionName, final String tmpCollectionName) {
    final var tempCollection = mongoDatabase.getOrCreateNewCollection(tmpCollectionName);
    final var collection = mongoDatabase.getOrCreateNewCollection(collectionName);

    final int BATCH_SIZE = 1000;
    final List<Document> batch = new ArrayList<>();

    try (final MongoCursor<Document> cursor = tempCollection.find().projection(excludeId()).iterator()) {
        while (cursor.hasNext()) {
            batch.add(cursor.next());

            if (batch.size() == BATCH_SIZE) {
                collection.insertMany(new ArrayList<>(batch));
                batch.clear();
            }
        }

        if (!batch.isEmpty()) {
            collection.insertMany(batch);
        }
    }
}

If you manage your own deployment and can build custom connector images, patching this method locally is the primary workaround suggested in the issue discussion [destination-mongodb issue]. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.