Hello team, am hoping someone can throw some light on this behaviour I encountered whilst configuring a Snowflake to Redshift connector. So, we made schema changes (add & remove columns) in the Snowflake tables we are trying to bring into Redshift as part of a migration. However, irrespective of the number of times we refreshed the schema from within the Airbyte connector it wasn’t picking up the underlying changes we had affected in the Snowflake database objects. We finally dropped and recreated the connector and this time around the schema picked up by the connector reflected the state in Snowflake. Is this by design or is there something I’m missing in terms of my understanding of the connector? Appreciate any thoughts from the community. Redshift destination connector version -

0.3.47

Snowflake source connector version -

0.1.18

Thanks

I saw an instance on Snowflake destination where destination namespace changed after a schema reset. Is your destination schema / namespace as expected?

Thanks @Lucas Wiley We created the Snowflake source before the schema changes and haven’t started the sync into Redshift yet. Luckily we caught this behaviour before we setup up a connection We are in the middle of development and so expect schema changes to keep happening Thinking if using Airbyte in this instance is going to cause unseen issues

Just reread your message. I think this is a common feature for many integration tools because they don't want to make assumptions about schema changes so they never drop a column. They just stop populating it until you trigger a full re sync

That makes perfect sense to keep the dropped column, but the connector wasn’t picking up the new columns either which is what I’m surprised about 🙂