Hi everyone, I need some help with an issue after ...
# replication-ask-ai
s
Hi everyone, I need some help with an issue after upgrading the Postgres destination to v3.0.0. I upgraded only the destination connector, and after the upgrade Airbyte stopped writing into my existing table. Even though my sync mode is still Incremental | Append, Airbyte is now creating a new table instead of appending to the old one. No schema changes were made on my side. What I’m seeing: Old table: still exists with the same column structure After upgrade, Airbyte created a new table with a different internal naming pattern Data is going into the new table, not the original one _airbyte_extracted_at, _airbyte_raw_id, and _airbyte_meta structure looks the same Only change was upgrading to Postgres Destination 3.0 Questions: Is this expected behavior with the new “Direct Load” architecture in v3.0? How can I force Airbyte to continue appending into my original destination table? Is there a recommended migration path to avoid table recreation during Postgres 3.0 upgrade? I can share table names, logs, and the connection ID if needed. Thanks in advance for any help! 🙏
k
Short answers: 1. Yes, some table behavior changes are expected with Direct Load in Postgres 3.0.0. 2. There is no documented way to “force” v3.0.0 to keep writing into the exact same pre‑3.0 table; you should treat this as a one‑time migration and point downstream logic at the new table. 3. The recommended migration path is to upgrade and then adjust downstream models/queries, not to preserve the old table name. Here’s what the docs say and how it maps to what you’re seeing: ### 1. Is this expected with Direct Load? Postgres 3.0.0 switches the destination to Direct Load: data is written directly to final tables, and raw tables are no longer used by default. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Postgres dest](https://docs.airbyte.com/integrations/destinations/postgres)] Key changes: - “No more raw tables by default: Data is now written directly to final tables.” [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300)] - Final tables now include
_airbyte_meta
and
_airbyte_generation_id
instead of
_airbyte_loaded_at
. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] The migration guide also notes that if you depended on both raw and final tables, you now need two destinations because the connector no longer supports writing both raw and final tables in a single connection under the direct‑load paradigm. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Direct load – what changes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/direct-load-tables#what-will-change)] The docs do not explicitly describe the exact naming pattern you’re seeing, but they do describe this as a breaking change in how final tables are written. Given that: - You upgraded the destination only. - Final‑table behavior and metadata columns changed. - Direct‑load implementations may need to create/alter tables for compatibility and schema evolution. [[Direct load – what changes](https://docs.airbyte.com/platform/using-airbyte/core-concepts/direct-load-tables#what-will-change)] It is consistent with the docs that upgrading to 3.0.0 could result in Airbyte creating a new final table and writing there rather than continuing to use the legacy one. ### 2. Forcing writes into the original table The documentation does not describe any configuration that guarantees “keep using the exact old table name and definition” after upgrading to 3.0.0. What it does say: - If you only use final tables, “no action required” and syncs “automatically use the new Direct Load architecture,” but it assumes you’ll adapt downstream logic to the new behavior/columns. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300)] - If you only depend on raw tables, you can enable “Raw tables only” to keep getting legacy‑style raw tables. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Direct load – raw tables compat](https://docs.airbyte.com/platform/using-airbyte/core-concepts/direct-load-tables#raw-tables-compatibility-support)] There is no documented setting to: - Keep Direct Load enabled and - Force the connector to keep using the original pre‑3.0 final table name/definition. Given that limitation, the documented path is to: - Accept the new final table layout created by 3.0.0, and - Update your downstream dbt/SQL to read from that table instead. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300)] ### 3. Recommended migration path to avoid “breakage” The Postgres 3.0.0 migration guide recommends: - If you only use final tables: simply upgrade; then update any logic that referenced
_airbyte_loaded_at
or older metadata to the new
_airbyte_meta
/
_airbyte_generation_id
structure. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300)] - If you only depend on raw tables: turn on “Raw tables only” before upgrading, so behavior stays compatible for raw‑table workflows. [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Direct load – raw tables compat](https://docs.airbyte.com/platform/using-airbyte/core-concepts/direct-load-tables#raw-tables-compatibility-support)] - If you need both raw and final tables: use two destinations and dual connections (one with “Raw tables only”, one without). [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Direct load – raw tables compat](https://docs.airbyte.com/platform/using-airbyte/core-concepts/direct-load-tables#raw-tables-compatibility-support)] More generally for Destinations V2 and similar breaking changes, Airbyte’s guidance is to: - Upgrade the destination version. - Let the first sync do the migration work (this can be longer). - Then update downstream dbt/SQL models to match the new table structures, rather than trying to keep the old ones unchanged. [[Destinations V2 quick start](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2#quick-start-to-upgrading); [Upgrading downstream transformations](https://docs.airbyte.com/release_notes/upgrading_to_destinations_v2#updating-downstream-transformations)] The docs do not describe a pattern that guarantees “no new table, keep writing into exactly the same table name/shape” for this 3.0.0 upgrade. ### Practical next steps Based on the documentation: - Treat the newly created table as the authoritative Direct Load final table. - Point your dashboards / dbt / SQL to this new table and adjust for the metadata changes (
_airbyte_meta
,
_airbyte_generation_id
). [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300); [Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] - Once you confirm everything works and you don’t need the legacy table, you can optionally drop the old one (similar to how the docs recommend cleaning up old raw tables only after verification). [[Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#removing-old-raw-tables)] If you’d like, you can share the exact old/new table names and connection ID with Airbyte support; they can confirm whether what you’re seeing aligns with the internal naming changes for Direct Load. The docs, however, don’t provide more detail than the above.
s
Why isn't it not following naming as mentioned in connection
@kapa.ai why isn't following naming conventions
k
A message from kapa.ai
The docs only partially explain naming behavior; they don’t document the exact algorithm that decides when a new table name is used, so there is some unavoidable uncertainty here. What is documented for Postgres: - Identifiers must follow Postgres rules (start with letter/underscore; then letters, digits, underscores, `$`; max 63 bytes; longer names are truncated). [[Postgres naming](https://docs.airbyte.com/integrations/destinations/postgres#naming-conventions); [Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] - Airbyte: - Uses quoted identifiers for final tables and columns, preserving case and replacing special characters with
_
. [[Postgres naming](https://docs.airbyte.com/integrations/destinations/postgres#naming-conventions)] - In raw‑tables‑only (legacy) mode, uses unquoted identifiers, also replacing special chars with
_
. [[Postgres naming](https://docs.airbyte.com/integrations/destinations/postgres#naming-conventions)] - Truncates identifiers longer than 63 characters and may additionally modify column names if truncation causes collisions. [[Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] What is not documented: - The exact deterministic mapping from: - old Destinations V2 / pre‑3.0 final table names to - new Direct Load final table names in 3.0.0, especially when names are already close to the 63‑char limit or collide after truncation. - Any guarantee that a pre‑3.0 final table name will be reused instead of creating a new one after upgrading to Direct Load. So if by “following naming conventions” you mean: - “Why doesn’t 3.0.0 keep using the exact same table name as before?” The docs don’t state that this is guaranteed, and they explicitly call 3.0.0 a breaking change that introduces a new architecture and metadata schema. [[Postgres changelog](https://docs.airbyte.com/integrations/destinations/postgres#changelog); [Postgres migration](https://docs.airbyte.com/integrations/destinations/postgres-migrations#upgrading-to-300)] - “Why is there a new internal‑looking suffix/prefix even though my stream name didn’t change?” The only relevant hint is that Airbyte may adjust/truncate identifiers (and then further modify them to avoid collisions), but the precise pattern is not documented. [[Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] Given the available information, the most accurate statement is: - Airbyte is following Postgres’ identifier rules (valid characters, quoting, 63‑byte limit, collision avoidance). [[Postgres naming](https://docs.airbyte.com/integrations/destinations/postgres#naming-conventions); [Schema map](https://docs.airbyte.com/integrations/destinations/postgres#schema-map)] - The exact naming transformation and why it picked this new name instead of reusing the old one is not described in the public docs, so I cannot explain the exact suffix/pattern from the knowledge sources. If you need to validate that what you see is correct for a specific connection, logs or table names would have to be checked against the connector code or by Airbyte support; that level of detail is not present in the documentation provided.