Hi Team I want to sync a few mysql database tables that are more than 50Gb in size to a clickhouse Data Warehouse. I do not want to move historic data and only want to sync new data upto a month old along with data that will get added daily from the tables . Since Airbyte syncs entire data the first time, I want to know if the following approach is feasible. Is it possible to make airbyte sync incrementally from the latest record that was added to the destination database?

Hi. you need to use the api to update the source to dynamic dates

But my source is a MYSQL database. Does airbyte have an api where it allows for setting of dates for mysql?

Unfortunately in the case of mysql there is no way to do it with the api. There is a workaround though - create in the database a table that updates on daily basis with the smae data you need- this way you will get the data every day dynamically.

Oh okay thank you for your help. I thought of that, but wanted to avoid any dependencies with jobs running in the DB to create a daily data aggregation table. I have to import 5 to 6 tables which are all quite big.

Hello Daryl Thomas, it's been a while without an update from us. Are you still having problems or did you find a solution?