https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> what is the difference between raw ...
# ask-ai
o
@kapa.ai what is the difference between raw and raw_stream tables in airbyte?
k
Based on the provided knowledge sources, Airbyte uses "raw tables" to temporarily store data as it is replicated from the source before any normalization or transformation occurs. These tables typically have names like 
_airbyte_raw_<stream_name>
or, in Destinations V2 and newer, 
airbyte_internal.<schema>_raw__stream_<stream_name>
. The term "raw_stream" appears in the context of Destinations V2 and some destination connectors (such as Oracle and BigQuery), where the raw table is named with a double underscore and the word "stream" (e.g., 
airbyte_internal.default_raw__stream_<stream_name>
). This naming convention is used to distinguish these tables as the raw, unprocessed data for a specific stream. Key points: - "Raw tables" and "raw_stream tables" refer to the same concept: tables that store the unprocessed, JSON-formatted data as it is ingested from the source. - The difference is primarily in the naming convention and schema location, which has evolved with Airbyte's platform updates (e.g., Destinations V2). - In Destinations V2, raw tables are now placed in the 
airbyte_internal
schema and named with the pattern 
<schema>_raw__stream_<stream_name>
, whereas previously they might have been named 
_airbyte_raw_<stream_name>
and located in the default schema. - The structure of these tables is similar: they contain columns like 
_airbyte_raw_id
, 
_airbyte_data
, 
_airbyte_extracted_at
, and 
_airbyte_loaded_at
(column names may vary slightly depending on Airbyte version and destination) [Upgrading to Destinations V2](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2), [Oracle Migration Guide](https://docs.airbyte.com/integrations/destinations/oracle-migrations#upgrading-to-100). In summary, "raw" and "raw_stream" tables in Airbyte serve the same purpose, with the difference being the naming convention and schema organization introduced in newer Airbyte versions. There is no functional difference between them beyond this naming and organizational change.
o
@kapa.ai in my case when I connect to powerbi both tables exist the raw_<stream_name> and the raw__stream_<stream_name> is that normal
k
A message from kapa.ai
It is possible for both 
raw_<stream_name>
and 
raw__stream_<stream_name>
tables to exist in your destination, especially if you have recently upgraded Airbyte or changed your destination connector version. According to the [Oracle Migration Guide](https://docs.airbyte.com/integrations/destinations/oracle-migrations#upgrading-to-100), when upgrading to Destinations V2, Airbyte changes the schema and naming convention for raw tables (from the old 
_airbyte_raw_<stream_name>
to the new 
<schema>_raw__stream_<stream_name>
format). The upgrade process does not delete or migrate your old raw tables—both the old and new tables can coexist unless you manually clean up the old ones. So, seeing both table types in Power BI is normal if you have migrated or upgraded your Airbyte setup. You should use the new 
raw__stream_<stream_name>
tables going forward, as the old ones are no longer updated after the migration. If you no longer need the old tables, you can safely remove them after verifying your data migration is complete and correct. Summary: Yes, it is normal to see both 
raw_<stream_name>
and 
raw__stream_<stream_name>
tables after an upgrade or migration, as Airbyte does not automatically delete or migrate old raw tables to the new format [Oracle Migration Guide](https://docs.airbyte.com/integrations/destinations/oracle-migrations#upgrading-to-100).
o
@kapa.ai could it be possible that the final duplicated table is connected to the <schema>_raw__stream_<stream_name> table while the SCD table is still connected to the old _airbyte_raw_<stream_name> and that is why my SCD table doesn't update correctly while my final deduplicated table is correct and up to date
k
A message from kapa.ai
Based on the knowledge sources, it is possible for both the old-style raw tables (e.g., 
_airbyte_raw_<stream_name>
) and the new Destinations V2 raw tables (e.g., 
<schema>_raw__stream_<stream_name>
) to coexist after an upgrade. However, after upgrading to Destinations V2, **Airbyte only updates the new raw tables**; the old raw tables are left untouched and are no longer kept up to date. Pre-existing SCD tables and unnested tables are also left untouched and are not updated by new syncs after the upgrade. Only the new final tables (deduplicated tables) are kept up to date by Airbyte syncs after the upgrade [Quick Start to Upgrading](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2#quick-start-to-upgrading). If your SCD table is still connected (via downstream models or queries) to the old 
_airbyte_raw_<stream_name>
table, it will not receive new data, which could explain why your SCD table is not updating correctly, while your final deduplicated table (which is connected to the new raw table) is up to date. This scenario is consistent with the documented upgrade process, where old tables are not automatically migrated or updated, and downstream models may require updates to reference the new table structure and names [Upgrading Downstream Transformations](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2#updating-downstream-transformations). In summary: Yes, it is possible that your SCD table is not updating because it is still connected to the old raw table, which is no longer updated after the upgrade, while your final deduplicated table is connected to the new raw table and is up to date. You should update your downstream models to reference the new raw table structure to ensure all tables are kept current.
Open in Slack
PreviousNext
Powered by