I have a connection with the schema change setting...
# replication-ask-ai
s
I have a connection with the schema change settings set to "propagate column changes only". However it has detected a new column but has not propagated the change. It is asking me to review the changes. Is this expected behaviour?
k
Yes, this is the expected behavior. Even though you have set the schema change settings to "propagate column changes only", Airbyte still allows you to review the changes before they are propagated. This is to ensure that you are aware of the changes and agree with them before they are implemented. Here are the steps to review non-breaking schema changes: 1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click Connections and select the connection with non-breaking changes (indicated by a yellow exclamation mark icon). 2. Click Review changes. 3. The Refreshed source schema dialog displays the changes detected. 4. Review the changes and click OK to close the dialog. 5. Scroll to the bottom of the page and click Save changes. By default, Airbyte ignores non-breaking changes and continues syncing. You can configure how Airbyte handles syncs when it detects non-breaking changes by [editing the stream configuration](https://docs.airbyte.com/cloud/managing-airbyte-cloud/edit-stream-configuration). For more details, you can refer to the [Manage schema changes](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes) section in the Airbyte documentation.
s
@kapa.ai OK - so if i save changes on this review it will add the column without needing to reset the data?
k
Yes, if you are reviewing non-breaking schema changes, you can save the changes without needing to reset the data. Here are the steps to do so: 1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click Connections and select the connection with non-breaking changes (indicated by a yellow exclamation mark icon). 2. Click Review changes. 3. The Refreshed source schema dialog displays the changes detected. 4. Review the changes and click OK to close the dialog. 5. Scroll to the bottom of the page and click Save changes. By default, Airbyte ignores non-breaking changes and continues syncing. You can configure how Airbyte handles syncs when it detects non-breaking changes by [editing the stream configuration](https://docs.airbyte.com/cloud/managing-airbyte-cloud/edit-stream-configuration). However, if there are breaking changes, such as columns that have been removed or changed, Airbyte recommends that you reset streams when you save the changes. A reset will delete data in the destination of the affected streams and then re-sync that data. Skipping a reset is discouraged and might lead to unexpected behavior ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/configuring-connections)). Please note that even if no source data is modified, a copy of the entire source table is appended onto the destination tables ([source](https://airbyte.com/tutorials/full-data-synchronization)).
s
@kapa.ai does the removal or addition of a column always require a reset, even with the 'propagate column changes only' setting enabled?
k
A message from kapa.ai
No, the removal or addition of a column does not always require a reset, even with the 'propagate column changes only' setting enabled. When a new column is detected and propagated, values for that column will be filled in for the updated rows. If you are missing values for rows not updated, a backfill can be done by completing a full refresh. When a column is deleted, the values for that column will stop updating for the updated rows and be filled with Null values ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes)). However, if a breaking change is detected, such as the removal of an existing primary key or cursor from the source, the connection will be paused for manual review to prevent future syncs from failing. In such cases, you might need to reset the data ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes)). Please note that resetting data will delete all records in your destination tables or files, but it will not delete the destination tables or files themselves ([source](https://docs.airbyte.com/operator-guides/reset)).
s
@kapa.ai Ok. The changes that I want to apply are: remove 1 column and add 1 column. Neither of these columns are a primary key or cursor field. So I can save changes, choose not to reset the data, and it will propagate those schema changes as described in your last message?
k
A message from kapa.ai
Yes, you can apply the changes you mentioned without resetting the data. Here's how you can do it: 1. On the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard, click Connections and select the connection where you want to apply the changes. 2. Click Review changes. The Refreshed source schema dialog will display the detected changes. 3. Review the changes and click OK to close the dialog. 4. Scroll to the bottom of the page and click Save changes. When you remove a column, the values for that column will stop updating for the updated rows and be filled with Null values. When you add a column, values for that column will be filled in for the updated rows. If you are missing values for rows not updated, a backfill can be done by completing a full refresh ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes)). Remember, these steps apply as long as the columns you are adding or removing are not primary keys or cursor fields. If they are, it would be considered a breaking change and the connection will be paused for manual review ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes)).
s
@kapa.ai I saved the changes but opted not to reset the stream because the changes were non-breaking. However on the next sync it failed, I got a bigquery error saying the provided schema does not match the table. So the change was not propagated as expected.
k
A message from kapa.ai
I'm sorry to hear that you're experiencing issues. The error message you're seeing suggests that the schema of your source data doesn't match the schema of your destination table in BigQuery. This could happen if the changes in the schema were not propagated as expected. When you add or remove columns without resetting the data, Airbyte should handle these non-breaking changes by filling in new column values for updated rows and filling removed column values with Null for updated rows. However, it seems like this did not happen as expected in your case. Here are a few steps you can take to troubleshoot this issue: 1. Check the logs: The logs provide detailed information about what's happening behind the scenes, and they can help pinpoint the root cause of the problem ([source](https://docs.airbyte.com/troubleshooting)). 2. Review the changes again: Go back to the Connections on the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard and review the changes you made. Make sure the changes were saved correctly. 3. Reach out to the community: If you're still unable to resolve the issue, consider reaching out to the Airbyte community. The community forum is a great place to ask for help, share your experiences, and learn from others who have faced similar issues ([source](https://docs.airbyte.com/troubleshooting)). 4. Open a Github ticket: If you're still unable to resolve the issue after reaching out to the community, it's time to open a support ticket. The Airbyte support team is here to help you with any issues you're facing with Airbyte ([source](https://docs.airbyte.com/troubleshooting)). Remember, it's always a good idea to back up your data and configurations before making changes to your schema or resetting your data.