@kapa.ai my mongo source has lot of data more than 100 gb so when at the sync entire 100 gb will be transfered to destination at once or it will be done in batches

Based on the knowledge sources provided, I can answer your question about how Airbyte handles large data transfers from a MongoDB source. When syncing large amounts of data from MongoDB, Airbyte does not transfer the entire dataset at once. Instead, it uses a batching mechanism to handle large data volumes efficiently. Here are the key points: 1. Batched Reading: The MongoDB source connector reads data in batches. As mentioned in the [Airbyte forum post](https://discuss.airbyte.io/t/2214), "sources read data in batches. It's not possible to know what the size of these batches are before reading them." 2. Checkpointing: Airbyte uses a checkpointing mechanism for reliable replication of large collections. This is mentioned in the [MongoDB V2 source documentation](https://docs.airbyte.com/integrations/sources/mongodb-v2#mongo-db): "Reliable replication of any collection size with checkpointing and chunking of data reads." 3. Incremental Sync: For subsequent syncs after the initial one, Airbyte supports incremental sync modes, which only replicate data that has been inserted or updated since the previous sync. This is more efficient for large datasets that change frequently. 4. CDC Support: The MongoDB connector supports Change Data Capture (CDC), which can be more efficient for keeping large datasets up to date after the initial sync. It's worth noting that while batching and checkpointing help manage large data transfers, the initial sync of a very large dataset can still take a considerable amount of time. The exact behavior may depend on factors such as your specific Airbyte configuration, network conditions, and the performance of your source and destination systems. If you're concerned about the performance of syncing such a large dataset, you might want to consider strategies like: - Using incremental sync modes where possible - Leveraging CDC if your use case allows - Potentially breaking up very large collections into smaller, more manageable parts if feasible for your data model For more detailed information on configuring and optimizing your MongoDB source connector, you can refer to the [MongoDB V2 source documentation](https://docs.airbyte.com/integrations/sources/mongodb-v2#mongo-db).

@kapa.ai after reading in batches will it transfer evenry batch when it is read or it will send all batches at once