I'm currently experiencing significant performance issues with the MongoDB CDC 1.0.3 connector when syncing data to a MongoDB destination (version 0.2). The synchronization process is notably slow, especially when dealing with large datasets; the connector reads data in batches of 5,000, but I have millions of records to handle. I'm seeking recommendations, fixes, or enhancements that the community might suggest to optimize this data synchronization. Has anyone encountered a similar situation and found effective ways to speed up the sync process? Any insights, best practices, or configuration tweaks that could enhance the performance would be immensely helpful.

Based on the information provided in the extracts, there are several ways you can optimize the performance of your MongoDB CDC connector when syncing large datasets. 1. Upgrade to MongoDB V2 Source Connector 1.0.0: The [1.0.0 version of the MongoDB V2 source connector](https://docs.airbyte.com/integrations/sources/mongodb-v2) introduces several performance improvements. It leverages Change Data Capture (CDC) to enhance the performance and reliability of syncs. This version provides better error handling, incremental delivery of data, and improved reliability of large syncs via frequent checkpointing. However, please note that this version includes breaking changes from previous versions of the connector, and you will need to reconfigure your existing MongoDB V2 configured source connectors. 2. Use Change Data Capture (CDC) for Incremental Delivery: The MongoDB V2 source connector 1.0.0 supports the use of [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc) for incremental delivery of changes. This can significantly improve the performance of data synchronization, especially for large datasets. 3. Frequent Checkpointing of Synced Data: The MongoDB V2 source connector 1.0.0 also supports frequent checkpointing of synced data. This can help improve the reliability of large syncs and prevent data loss in case of failures. 4. Use Incremental Sync Mode: When setting up your Airbyte connection, consider using the Incremental | Append sync mode. This mode only looks at the rows that have changed in the source and syncs them to the destination, which can be more efficient than syncing the whole source table. Note that when using CDC, you don't need to provide a Cursor field as changes in the source are detected via the Debezium connector stream ([source](https://airbyte.com/tutorials/mysql-change-data-capture-cdc)). 5. Set Frequent Syncs for CDC: To prevent the additional data from logical replication from filling up your disk space, set frequent syncs for CDC. This ensures that the data doesn't stay on your disk for too long ([source](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting)). Remember to test these changes in a controlled environment before deploying them to your production system. If you're still experiencing performance issues after trying these suggestions, you might want to reach out to the Airbyte community for further assistance.

I am already using the latest version of source mongodb connector which is 1.0.3 and I working with CDC as well @kapa.ai

That's great! The MongoDB V2 source connector version 1.0.3 you're using leverages Change Data Capture (CDC) to improve the performance and reliability of syncs. This version provides better error handling, incremental delivery of data, and improved reliability of large syncs via frequent checkpointing [source](https://docs.airbyte.com/integrations/sources/mongodb-v2-migrations). Here are some key features and considerations when working with CDC: - CDC is used in conjunction with various Sync modes that Airbyte supports, such as incremental dedupe synchronization [source](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication). - CDC incremental replication reads a log of the changes that have been made to the source database and transmits these changes to the destination. This means it's not necessary for the source data to have a suitable cursor field [source](https://airbyte.com/tutorials/incremental-change-data-capture-cdc-replication). - CDC incremental is only supported for tables with primary keys. Modifications must be made using `DELETE`/`INSERT`/`UPDATE`. Changes made from `TRUNCATE`/`ALTER` won't appear in logs and therefore in your destination [source](https://docs.airbyte.com/understanding-airbyte/cdc). - When setting up a connection for CDC, you can set the replication frequency to as frequently as every 5 minutes. For the sync mode, you should use Incremental | Append mode to only look at the rows that have changed in the source and sync them to the destination [source](https://airbyte.com/tutorials/mysql-change-data-capture-cdc). Remember to review the [MongoDB V2 source connector documentation](https://docs.airbyte.com/integrations/sources/mongodb-v2) for more details on how to use it effectively.

@kapa.ai can i enhance the reading or writing other than increase the frequency of sync

A message from kapa.ai