https://linen.dev logo
r

ruslan

05/16/2021, 7:29 PM
Hey folks. Just curious, what is the rationale for having source as independent entity in the API ? I assume you can't reuse same source (because it has specific credentials/settings) in multiple connections, but you can share destinations.
u

user

05/17/2021, 12:55 AM
Hi Ruslan, the source can also be reused in multiple connections. A common thing we see is seeing up multiple connections between the same source/destination each syncing different parts of the source database
u

user

05/17/2021, 9:47 AM
@Davin Chia (Airbyte) Thank you, Davin. That makes sense. But how do you handle situation when source can't be shared ? For instance in case of PSQL WAL you need to tell DB to release consumed portion of the transaction log. So if "connection" (pipeline in my terminology) releases WAL until LSN 100 but connectio2 wants to receive data from LSN 20 it will have to resync entire table.
u

user

05/17/2021, 9:51 AM
Ah I see what you mean. That is a good question and I don't know enough to answer you. @Jared Rhizor (Airbyte) what do you think?
u

user

05/17/2021, 3:57 PM
Right now I think only CDC has sources that contain configuration that is really sync-specific. It’s arguable that we should offer the ability to configure some source-specific information (like the CDC info) at the sync level instead of the source level to handle cases like this so the source is always truly reusable.
u

user

05/17/2021, 3:57 PM
Have you run into this outside of CDC?
u

user

05/17/2021, 3:58 PM
Also, would you mind creating a ticket to be able to reuse CDC sources?