@kapa.ai, How can I move the destination schema used while keeping all the history/etc?

Moving to a New Destination Schema While Preserving History The recommended approach for this is dual-writing, which lets you migrate to a new schema without losing historical data. Here's how to do it: Steps 1. Create a new destination pointing to the same database, but with a different

Default Schema

to avoid collisions with your existing tables. 2. Create a new connection using your existing source and the newly created destination. Match all settings from your pre-existing connection. 3. Copy existing raw data to the new schema (required for incremental streams to avoid re-syncing all history). Run SQL like the following in your data warehouse for each incremental stream:INSERT INTO new_schema.airbyte_internal.stream_raw_table SELECT * FROM old_schema.airbyte_internal.stream_raw_table;The exact SQL will vary by destination (BigQuery, Snowflake, Redshift, Postgres). [Additional Steps for Incremental Syncs] 4. Copy the connection state from your existing connection to the new one:This ensures Airbyte won't re-sync historical data from scratch. ◦ Go to the old connection → Settings → Advanced → copy the state. ◦ Go to the new connection → paste the state → click Update State. 1. Enable the new connection. It will now write to the new schema with all history intact and continue syncing new data in parallel. 2. Once you've validated the new schema, update your dashboards/downstream models to point to the new tables, then pause the old connection. [Dual-Writing Guide] Key Notes • This approach works for both full refresh and incremental sync modes, though the state-copying step is only needed for incremental streams. • Pre-existing raw tables in the old schema will remain untouched — you can delete them at your convenience after the migration. • Each destination version is managed separately, so if you have multiple destinations, handle them one by one. [Quick Start to Upgrading] Create a new destination Copy connection state 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai I have a Mysql->Snowflake connection, the snowflake destination connector is using direct-load to raw tables (with a 'PUBLIC' namespace), and we want to have it writing final tables in a 'SNAPSHOTS' schema

Moving from Raw-Tables-Only to Final Tables in a 'SNAPSHOTS' Schema Based on the knowledge sources, your current setup uses the Legacy raw tables option (equivalent to the old "Disable Final Tables"), which writes only raw tables. You want to switch to writing final tables into a

SNAPSHOTS

schema. Here's what you need to know: Important Limitation > The connector no longer supports writing both the raw and final tables in a single connection. [Snowflake Migration Guide] So you cannot have the same destination connector write to both raw tables and final tables simultaneously. Recommended Approach: Dual-Write Migration 1. Create a new Snowflake destination with: ◦ The same database/credentials as your existing destination ◦

Default Schema

set to

SNAPSHOTS

◦ Do not enable the

Legacy raw tables

/

Disable Final Tables

option — this will make it write final tables 1. Create a new connection from your MySQL source to this new destination, matching the settings of your existing connection. 2. Copy the incremental stream state from your old connection to the new one: ◦ Old connection → Settings → Advanced → copy the state ◦ New connection → paste the state → Update State ◦ This prevents a full re-sync of historical data 1. Copy existing raw data to the new schema for any incremental streams, so history is preserved without re-syncing from the source. [Additional Steps for Incremental Syncs] 2. Enable the new connection — it will now write final tables into

SNAPSHOTS

. 3. Once validated, pause the old connection. Your old raw tables in

PUBLIC

will remain untouched and can be dropped at your convenience. [Dual-Writing Guide] Note on Schema Evolution The Snowflake 4.0.0 direct-load connector requires

ALTER TABLE

permissions to support schema evolution. Make sure your Snowflake user has this permission granted before the new connection starts syncing. [Snowflake Migration Guide] Copy connection state 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.