Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

<@U01AB6V6NMQ> have some architectural questions.
1. How does Airbyte handle the transfer of data between a source and target connector.
2. Do connector support pulling data from a sharded source (postgres or Kafka partitions) --- if so, how does it get moved to the same destination table?

1. for this question you can read our docs:
    a. <https://docs.airbyte.io/understanding-airbyte/high-level-view>
    b. <https://docs.airbyte.io/understanding-airbyte/airbyte-specification>
2. you want to bring data from different source and save in the same destination table? If you only want to bring from a shared source is not a problem because you can connect to a cluster and not the specific replica.

thanks <@U01MMSDJGC9> re pt 1.  I have seen the docs, but those are on a different level I guess. I want to understand e.g. do the source and destination connector have to be on the same machine for transfer to occur? if not what is the IPC mechanism between the two ?

re pt2. okay, so for a sharded source, a single connector can pull from the cluster --- but then how does it make use of the parallelism offered by such sharding?

pt 2 can also be split between the above case (single clustered source) and a set of different data sources with *same* schema but in physically different locations.