Im migrating from mssql to postgres , mssql connector v5 and postgres v2 , but after loading the data in to airbyte internal raw table its taking too much time to load into final table what is the issue?

Slow Loading from Raw Tables to Final Tables (MSSQL → Postgres) This is a known performance challenge when using PostgreSQL as a destination, especially with large datasets. Here are the key issues and recommendations based on the available information: Root Causes 1.

_airbyte_meta

column computation: The SQL query that loads data from the raw table into the final table includes computation of the

_airbyte_meta

column, which can be extremely slow for large datasets. One user reported that removing

_airbyte_meta

from the query made it complete successfully. [Postgres loading issue] 2. Postgres is not a data warehouse: Airbyte explicitly recommends using Postgres only for small data volumes (under ~10GB). For larger volumes, it is likely to perform poorly, especially during typing and deduplication. [Postgres limitations] 3. Redundant indexes on raw tables: Older versions of the destination-postgres connector created duplicate indexes on

_airbyte_raw_id

, significantly slowing down inserts. This was reported to cause ~50% slowdown. [Slow inserts issue] Recommendations 1. Upgrade the destination-postgres connector to v3.x: Version 3.0.0 introduces Direct Load architecture, which writes data directly to final tables without intermediate raw tables, eliminating the T+D (typing & deduplication) query overhead entirely. [Postgres migration guide] 2. Tune your Postgres configuration: One user resolved a similar issue by adjusting `postgresql.conf`: [Postgres loading issue]work_mem = 72MB client_connection_check_interval = 30000 tcp_keepalives_idle = 10 3. Scale IOPS on your Postgres instance: When dealing with large data volumes, disk throughput (IOPS) is as important as memory and CPU. [Postgres limitations] 4. Consider a proper data warehouse: If your dataset is large (>10GB), Airbyte recommends using BigQuery, Snowflake, or Redshift instead of Postgres as the destination. [Postgres limitations] Quick Summary The most impactful fix would be upgrading to destination-postgres v3.x, which avoids the raw-to-final table loading step entirely. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help., with interactive elements

Which airbyte support this new direct load ?