<@U04S8PLJWMT> I am upgrading my PostgresQL data d...
# replication-ask-ai
q
@kapa.ai I am upgrading my PostgresQL data destination to version 3. Will destination tables be dropped and recreated as part of the upgrade process? If so, will data be repopulated from the raw tables, the existing destination tables, or fully reloaded from sources?
k
When upgrading the Postgres destination to version 3.0.0, here's what you need to know about table handling: What Happens to Your Tables Existing raw tables will NOT be automatically dropped or deleted. According to the migration guide, existing raw tables are preserved during the upgrade — new syncs simply won't write to them unless you explicitly enable the "Raw tables only" option. [Postgres Migration Guide] For final (destination) tables, the behavior depends on your situation: • If you only use final tables: No action is required. Your syncs will automatically use the new Direct Load architecture, writing directly to final tables without intermediate raw tables. [Postgres Migration Guide] • If you depend on both raw and final tables: This combination is no longer supported in a single destination. You would need to configure two separate Postgres destinations. [Postgres Migration Guide] Key Schema Changes The new version introduces changes to metadata columns in final tables: •
_airbyte_meta
(JSONB) and
_airbyte_generation_id
(BIGINT) columns are added. • The
_airbyte_loaded_at
column is removed. If you have downstream dbt models or SQL queries referencing
_airbyte_loaded_at
or raw tables, you'll need to update them. [Postgres Migration Guide] Data Repopulation The knowledge sources do not explicitly describe a full drop-and-recreate of final tables as part of the 3.0.0 upgrade itself. However, Airbyte may perform a "soft reset" (drop and recreate the final table) in certain situations — such as schema migrations — and in those cases, data is repopulated from the raw tables, not from the source. [Typing and deduping] Since version 3.0.0 removes raw tables by default, if a soft reset were triggered post-upgrade and you had already removed your old raw tables, repopulation from raw tables would not be possible. It is therefore advisable to verify your syncs are working correctly before removing old raw tables. [Postgres Migration Guide] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai So if I have a view in my PostgresQL database that queries the final destination tables, do I need to drop that view before upgrading?
k
Based on the knowledge sources, you don't necessarily need to drop the view before upgrading, but you need to be aware of an important risk and a configuration option. Views on Final Tables and CASCADE Drops The Postgres destination connector sometimes needs to recreate final tables (e.g., during schema migrations or refreshes). If you have a view depending on a final table, Postgres will require the connector to use
DROP ... CASCADE
to drop the table. This will also drop your dependent view. [Postgres destination] To handle this, the connector has a "Drop tables with CASCADE" option. If this is enabled, your dependent view will be dropped automatically whenever the connector recreates a table. The documentation strongly recommends: • Using a tool like
dbt
to automate the recreation of dependent objects. • Using an orchestrator to trigger
dbt
after syncs, so the view is reliably recreated. > ⚠️ Permanent data loss is possible if you enable CASCADE without a way to recreate dependent objects. [Postgres destination] Practical Recommendation Rather than dropping the view manually before upgrading, the safer approach is to: 1. Have an automated way to recreate the view (e.g., via
dbt
). 2. Enable the "Drop tables with CASCADE" option so the connector doesn't fail when it needs to recreate tables. 3. Ensure your orchestration triggers the view recreation after each sync where a table rebuild might occur. The knowledge sources do not explicitly state whether the 3.0.0 upgrade itself will trigger a table drop/recreate for all connections, so caution is warranted. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
q
@kapa.ai I've upgraded my PostgresQL version to 3.0 but my final tables still have the airbyte_extracted_at field. Is this expected? The raw tables are no longer updating so the upgrade seems to have been successful, but based on the documentation I expected the airbyte_extracted_at field to be dropped.
k
New message text here