Hi, I have a question regarding the sync of data from a relational database. If more than one table should be synced how is a consistent snapshot created and/or kept/stored for all the tables? Or can they be out of sync?

each table is called a stream and with the new work on incremental, there will be a state file associated to each stream (each table) to save a "cursor" position of what has been replicated so far for each

Hi Chris, thanks for the explanation. Could you explain what is tracked in the “cursor” position? Do you keep track of this cursor on “record level”? Or on “table level”?

The cursor is (for the moment) the MAX value of a column (typically a date like updated_at) so it records for the table the latest updated_at we've seen so far, so next time a sync is run, it can easily check if new rows were inserted since previous run by doing something like

SELECT * FRON table WHERE updated_at > <last_recorded_updated_at>

Ah, well I guess then sometimes you will miss updates. This due to the fact that you probably are not reading data in a full “read consistent state” during the complete sync session. A second issue is that if you sync several tables you probably won’t get a “read consistent snapshot” for all the tables. I ran into all these issues when syncing data. Especially on data sources which are constantly changing (no stale window to fetch the data). The only solution which worked was true change data capture (which sometimes is cumbersome to implement/make robust as well).

Yes, Change Data Capture will come next in our roadmap but we have to slowly implement each method before getting there to handle most cases (when a source does not provide binary logs for changes for example)

Agree.