Incremental sync and initial data in destination table: • I have a huge table in BigQuery which I synced using Google tools (one off). • Currently, the maximal value of the column

last_update_dt

is X • Now, I want to define an incremental sync to that table. The sync will be based on

last_update_dt

column. • I want the first sync that runs to start from records where

last_update_dt > X

i.e. I want the incremental sync to pick up from the current state of the destination table (vs. from scratch). Is that possible? [edit] Is it possible to do something like set the connection state in the postgres db to point to the latest value of the column?

Hi Assaf, it should be possible to set the connection state but I need to look further into this!

@Nataly Merezhuk (Airbyte) thanks I managed to do it manually. Not very nice and not very safe, but it seemed to work. • Create a connection with a single large incremental table • Start sync • Stop sync after few seconds • Reset state • in postgres;

select id from state where connection_id=<your connection id>

And then

update state set type='STREAM', namespace=<namespace> stream_name=<stream_name> state='{"cursor": <max id that you synced manually>, "stream_name": <stream_name> "cursor_field": [<your cursor field e.g "id">], "stream_namespace": <namespace> "cursor_record_count": 1}',  where id=<id of record from prev step>

Thanks so much for the update!

Hi @Assaf Pinhasi, thanks for sharing this. I've the same requirement as you and tried to replicate your solution. However, it appears to run a full sync (after completing these steps) it ignores the JSON values that I entered into State. The sync mode that I tried was both Incremental (append) and Incremental (Deduped) Any idea what I'm missing?

not sure. I did this once, and it's possible that my example was indeed based on ids and not timestamps. Can't remember...