Hi! I'm trying to build a new source connector, and I have access to the transaction log, so it seems like a CDC connector is possible. Would that be recommended? There don't seem to be a ton of docs on building them. It would be nice to fully support deletes, but if new CDC connectors are difficult then I can fall back to soft deletes.

@Marcos Marx (Airbyte) turned this message into Zendesk ticket 2842 to ensure timely resolution!

Hi. Not sure I fully understand the question here. are you asking should you build a cdc and if it`s to hard to build soft delete? if yes - 1. it really depends on your needs - so if you need to build cdc then build it 2. hard is very subjective so... 3. i would create cdc or soft delete dependent on the data and needs

Hi! Yes that is my question. I would like to build CDC, and I cannot find any documentation on how to do so. Can you point me to some? The only reference I found at https://docs.airbyte.com/understanding-airbyte/cdc says "create a ticket if you need CDC support on another database", which indicates to me that the process isn't self-serve. I looked at existing CDC connectors (specifically postgres) and it's complex so I couldn't figure out how to copy the pattern. What methods need to be implemented? What kind of AirbyteMessages should be returned? Can I write it in python?

my answer is yes and no. yes: So actually you can add that logic the same way you create a custom source\destination- for example: (I'm giving a merge logic example since this is what airbyte creates behind the scenes for the incremental logic and using that saves logs etc...) google cloud bigquery - they dont really have a concept of a primary key but rather a merge key but it`s the same lady in a different shade. you need to create a DDL and DML statement the ddl will be "create or replace table..." (in some cases you only need this)- https://cloud.google.com/bigquery/docs/reference/standard-sql/data-definition-language and the dml will be "merge" - https://cloud.google.com/bigquery/docs/reference/standard-sql/dml-syntax#merge_statement this dwh has an easy way and built in way to setup a merge logic BUT!!!!- (this is the no ) like airbyte team said if you need this to other dbs please open a ticket and that is probably because it could be very difficult or maybe first time that you work with that db or maybe you dont need it right away - so that option is awesome to have!!! so to summaries - you can easily add this logic as an option when creating a custom source\ destination (have this option in the UI even) though this requires some (in some cases a lot of work) work I strongly recommend to open a ticket if you dont have experience in building cdc logic. - https://airbyte.com/blog/change-data-capture-definition-methods-and-benefits if you still want to do this - depending on the db- you need to fully understand the how to do the exact same work that airbyte does when creating cdc the doc you have is the logic that you need to create

Thanks for the links! Just to clarify, this is a connector for a database that I work on. Imagine that I work at MongoDB and i am building the first iteration of the MongoDB connector (it's not mongo but it's similar), and the goal is to send a PR to make it available for all airbyte users. So "the first time you work with that db" doesn't apply -- i know exactly how the db works and i can even modify it if needed. And "you dont need it right away" doesn't apply -- I don't need it at all personally, I'm trying to build the best connector so my company's customers can use it. Based on the few examples of CDC connectors like postgres and mysql, and several examples of databases that don't have CDC connectors like MongoDB, I think I'll stick with avoiding CDC for now. Thanks!

@Lee Danilek feel free to post more questions in the channel! We’d love to see this connector and get it merged into our main branch.

Ooh I figured out how to replicate deletes in incremental mode with a python connector! To each record returned from

read_records

add these three fields:

_ab_cdc_lsn

(timestamp as an integer),

_ab_cdc_updated_at

(timestamp as a string), and

_ab_cdc_deleted_at

(timestamp as a string, or None). Then add these keys to the json schema with types

{"type": "number"}

(even though it's an integer),

{"type": "string"}

, and

{"type": "string"}

(even though it can be null) respectively. I still don't know if there's documentation for this; I figured it out by digging into the Postgres connector code.

Thanks @Lee Danilek for reporting back after digging into the code. I don’t think it is documented but hopefully this helps other users in the meantime.