@kapa.ai, We have a MySql->Snowflake connection taking daily full-refresh/append snapshots with final tables disabled. We want to reconfigure it to Snowflake direct-load, unfortunately we do have some deprecated but still running processes using the raw tables. I know I can update the existing connection to direct-load with legacy raw tables and that will keep the raw tables updated without breaking the processes. Raw tables compatibility support says I should create a new connection that direct-loads to the final tables. If I do that, how can I migrate the current raw history to the new connection?

Thinking completed The knowledge sources do not provide an automated tool, migration script, or built-in Airbyte process for converting historical raw JSON records into typed direct-load final tables. However, based on how Airbyte handles dual-writing and raw table configurations, here is what you need to know and how you can approach this: 1. The Dual-Connection Strategy Because direct-load does not support writing both raw tables and final tables within a single destination configuration, your intended strategy is correct [what will change; upgrading to 400]: • Keep your current connection pointing to a Snowflake destination with Legacy raw tables enabled (or

Disable Final Tables

turned on) to ensure your deprecated downstream dependencies continue running uninterrupted [what will change; upgrading to 400]. • Create a second destination configured for Direct Load (with final tables enabled / legacy raw tables disabled) and run a separate connection from MySQL in parallel [what will change; upgrading to 400]. 2. Handling Historical Data in the New Direct-Load Connection Because your sync mode is Full Refresh | Append, each daily run will extract and append the full source table. Regarding historical snapshots: Option A: Backfill / Re-sync from the Source (Recommended if source has history) If your MySQL source retains the full historical log/data (or if you only require daily snapshots moving forward), the cleanest Airbyte-native approach is to let the new direct-load connection run its daily syncs. In

Append

mode, direct-load connectors insert records directly into the target table [what will change]. Option B: Manual SQL Extraction from Raw Tables to the Final Table If your MySQL source only contains current state and your Snowflake raw tables hold irreplaceable historical snapshot partitions (e.g., past

_airbyte_extracted_at

dates), Airbyte will not backfill those final tables from raw tables automatically [what will change]. You will need to manually extract and insert that data in Snowflake: 1. Let the new connection sync once: Run the new direct-load connection to create the final typed table with its appropriate schema and metadata columns (

_airbyte_raw_id

,

_airbyte_extracted_at

,

_airbyte_meta

,

_airbyte_generation_id

) [what will change; postgres migrations]. 2. Execute an

INSERT INTO ... SELECT

query in Snowflake: Parse the

_airbyte_data

JSON/VARIANT column from your legacy raw table, cast each field to the destination table's target column types, and insert the rows into your newly generated final table. _(Note: If your streams used an incremental mode, Airbyte allows copying connection state to prevent re-syncing [additional steps for incremental sync modes], but for Full Refresh modes, connection state transfer does not apply.)_ 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.