Does airbyte recoginize each partition reading from postgres as one table? If there is one partitioned table with 10 partitions, then airbytes sees 10 tables?

Based on the knowledge sources, this is a known behavior with Airbyte's Postgres source connector. Yes, Airbyte treats each partition as a separate table. When you have a partitioned table (e.g.,

messages

) with 10 daily partitions (e.g.,

messages_20220518

,

messages_20220519

, etc.), Airbyte will see and list each partition as an individual stream/table. [Forum Post] This has a key implication for CDC (incremental) syncs: • You can sync each individual partition (e.g.,

messages_20220518

) incrementally. • You cannot reliably sync the root partitioned table (

messages

) incrementally via CDC — it returns 0 rows/bytes even when using

publish_via_partition_root

in the publication definition. • The root table can be synced with Full Refresh, but not incrementally. [Forum Post] The suggested workarounds from the community discussions are: 1. Sync each partition individually in incremental mode (but this becomes unwieldy as new partitions are added daily). 2. Split into two connections: ◦ One connection for incremental tables using CDC. ◦ One connection for the partitioned root table using Full Refresh (without CDC). [Forum Post] This is a known limitation tracked as a GitHub issue, and as of the time of these discussions, no native fix had been implemented. You may want to check the Airbyte GitHub for the latest status on this issue. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.