Hello everyone, I'm currently facing an issue with the MongoDB source connector (CDC 1.0.3) in combination with BigQuery as the destination. The problem arises when new columns are added to a collection; they are not detected due to the limitation of the maximum sample size being 100k. This limitation creates challenges, especially in dynamic environments where schema changes are frequent. When new columns are introduced, the current setup fails to recognize these changes, leading to incomplete data synchronization. I am reaching out to the community to inquire if there are any workarounds or solutions to address this problem effectively. Any guidance or insights on how to handle dynamic schema changes in MongoDB and ensure accurate schema detection during the synchronization process would be greatly appreciated.

Hello, You're correct that MongoDB's dynamic nature can pose challenges for data synchronization, especially when new columns are added. Airbyte's MongoDB V2 source connector leverages Change Data Capture (CDC) to improve the performance and reliability of syncs, but it does have some limitations when it comes to schema changes [source](https://docs.airbyte.com/integrations/sources/mongodb-v2-migrations). Airbyte provides a few ways to manage schema changes. For each connection, you can specify how Airbyte should handle any change of schema in the source. The options include propagating all changes, propagating column changes only, ignoring any schema change, or pausing the connection once a change is detected [source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes). In addition to automatic schema change detection, you can manually refresh the source schema to stay up to date with changes in your schema. This can be done from the Airbyte Cloud dashboard by clicking on Connections, selecting the connection you want to refresh, clicking the Replication tab, and then clicking Refresh source schema [source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes). However, it's important to note that the current behavior of Incremental is not able to handle source schema changes yet, for example, when a column is added, renamed or deleted from an existing table etc. It is recommended to trigger a [Full refresh - Overwrite](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite) to correctly replicate the data to the destination with the new schema changes [source](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append). I hope this helps! If you have further questions, feel free to ask.

when also manually refreshing the schema the new column doesn't appear i believe it's because the new documents with the newly added column didn't fall in the sample scan so is there any workaround to that @kapa.ai

Hmm, I don't know enough to give you a confident answer yet. However, based on the information provided, Airbyte does have a feature to manually refresh the source schema to stay up to date with changes in your schema. Here are the steps to do that: 1. On the [Airbyte Cloud](http://cloud.airbyte.com) dashboard, click Connections and then click the connection you want to refresh. 2. Click the Replication tab. 3. In the Activate the streams you want to sync table, click Refresh source schema to fetch the schema of your data source. 4. If there are changes to the schema, you can review them in the Refreshed source schema dialog. If the new column still doesn't appear after manually refreshing the schema, it might be due to other factors not covered in the provided extracts. I would recommend checking out the [Airbyte documentation](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes/) or reaching out to the Airbyte community for further assistance.