Hello everyone!
I am using airbyte to extract data from a MySQL databse to S3 (Minio) using CDC and .parquet as output.
It works great and the api is so easy to use, thank you for that!
I was wondering if anyone know a tool for my need. When using CDC, I have a new file at each synchronisation when changes are detected. I'd like to know if there is a way to version data based on cdc extraction from s3? For example, if I have those files containing CDC extractions:
data-0.parquet
data-1.parquet
data-2.parquet
I want to rebuild a file based on the version 0,1 or 2.
I know this question is not related to Airbyte functionalities, but in case, if anyone has heard about a tool managing CDC extractions, it'll be so great! Thank you in advance, and sorry if this question is out of the box.