@kapa.ai I am upgrading airbyte from 0.63.8 to 2.0.0 It seems that in the 2.0.0 version and 4.0.38 version of Snowflake connector there is no raw data persistency in the airbyte internal schema before merging into the destination table. It's either I have only the raw table or only the destination table. This situation does not really allow migrating to new version, it also removes the visibility of whether the data is lost when we wrote it into snowflake or when we merged into destination. In the past I was able to find bugs just because I had this layer

You're correct in your observation. The Snowflake destination connector v4.0.0 introduced the Direct Load paradigm, which fundamentally changes how data is written compared to the previous Typing & Deduping (T+D) approach. [Snowflake Migration Guide] What changed Under the old T+D approach, data was first written to raw tables in

airbyte_internal

, then typed/deduped into final tables — giving you exactly the two-layer visibility you relied on. Under Direct Load, the connector offloads type-casting to itself and writes typed data directly to the final table, eliminating the need for persistent raw tables. [Direct Load Tables] Your options The connector does still support a "Legacy raw tables" mode, but it's an either/or choice — not both simultaneously: • Final tables only (default): Direct Load, no raw tables persisted. • Legacy raw tables only: Enable the

Legacy raw tables

option on the connector. This is equivalent to the old

Disable Final Tables

option — the connector will only write raw tables and will not create final tables at all. [Direct Load Tables] > The connector no longer supports writing both the raw and final tables in a single connection. Workaround for dual visibility If you need both raw and final tables (for the debugging/auditing layer you described), the supported approach is to configure two separate destination connectors and run two connections in parallel: 1. One connector with

Legacy raw tables

enabled → writes only raw tables. 2. One connector without it → writes only final tables (via Direct Load). [Snowflake Migration Guide; Direct Load Tables] This is a deliberate architectural trade-off Airbyte made to reduce warehouse compute costs and storage, but it does come at the cost of the intermediate visibility layer you previously relied on for debugging. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

but this would mean reading the data twice? why would I need such an approach? what good are the raw tables by themselves? @kapa.ai

You raise a fair point. The dual-connector workaround does mean syncing from your source twice, which doubles the load on your source system and increases costs. Regarding what raw tables are useful for on their own — the knowledge sources do mention a few legitimate use cases: • Auditing unexpected or misformatted data: Raw tables capture the original JSON payload before any type casting, so you can inspect what Airbyte actually received from the source. [Upgrading to Destinations V2] • Downstream transformations built on raw data: Some users build their own transformation pipelines directly on top of raw tables rather than using Airbyte's typed final tables. [Upgrading as Raw Table User] However, I want to be honest: the knowledge sources do not provide a strong justification for the dual-connector approach in your specific scenario (debugging data loss between ingestion and merge). Airbyte's own documentation acknowledges that raw tables are intended as a staging ground and explicitly warns: > "We cannot guarantee the same level of stability as for final tables in your destination schema, nor will features like error handling be implemented in the raw tables." [Upgrading as Raw Table User] The Direct Load architecture moves type-casting into the connector itself, so the intermediate layer you previously used to distinguish "data lost at write" vs "data lost at merge" no longer exists as a persistent artifact. The knowledge sources don't offer an alternative solution to recover that specific debugging capability within a single connection. This appears to be a deliberate architectural trade-off with no fully equivalent replacement in the current design. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

@kapa.ai complete nonsense...with the old approach I cope with airbyte's inability to quickly extract data by creating my own process and writing into the raw table, then airbyte would pick it up and merge. But now, this is not possible. There is no place for me to stage the data

Your frustration is completely understandable. You're describing a legitimate and clever workflow: writing data into Airbyte's raw tables yourself as a staging area, then letting Airbyte's T+D process pick it up and merge it into the final typed table. That was a valid pattern under the old architecture. Unfortunately, based on the knowledge sources available, I have to confirm that this workflow is no longer supported under Direct Load. The raw tables are no longer a persistent, writable staging layer — they either don't exist at all (default Direct Load mode) or exist only as the sole output (Legacy raw tables mode, with no final table). [Direct Load Tables] The knowledge sources do not describe any alternative staging mechanism that would replicate what you were doing. Direct Load writes typed data directly to the final table, bypassing any intermediate persistent layer entirely. [Direct Load Tables] Given this, your realistic options based on what the documentation states are limited: 1. Stay on Legacy raw tables mode — but then you lose the final typed table and have to handle merging yourself entirely. 2. Use the dual-connector workaround — which you've already correctly identified as inefficient. 3. Delay upgrading until Airbyte potentially introduces a solution that fits your use case. I'd recommend raising this as a feature request or bug with Airbyte directly, as your use case (custom staging into raw tables) represents a real integration pattern that the current Direct Load architecture has broken without an equivalent replacement. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.