Hi, quick question. I created test table and put X number of rows. I synced and they went to destination. Now, If I add more rows to the table, either Airbyte does the full load again with full refresh/append option or it doesn't pickup the new records with incremental/append. Any idea?

Hi @Oscar Gonzalez it depends on the sync mode you've set up. If you've chose the incremental mode only new records (according to the cursor column/field) will be loaded in the next sync.

Can a table be pre-loaded? and then just use the incremental mode to get only the changes? or that wont work

Could you tell me why you'd like to preload and no go with an initial incremental load?

Because we have a table with 60 million and Airbyte will timeout before loading all. Now, we changed the parameters in the source tables as suggested but still failing. So we are testing if with a pre-load in the table can only send the difference if we add records but thats not working.

If you're able to preload the data yourself you can try to update the state of the incremental load in the Airbyte database to set it to the value at which you want the load to start, this is a hacky workaround that can work. Did you try loading the table the usual way? Is the timeout a concern or something you encountered initially?

So we've been experimenting a bit (Oscar and I are in the same company) to see if the connector smartly checks the diff between source/destination, but seems like each new connection is ignorant of whatever the source/destination have in common (i.e. starts fresh), is that the intended behavior?

The incremental load offers awareness of what data exists in the destination thanks to the storage of a state in the database. This state stores the value at which the last incremental sync stopped, so when a new one is triggered only the new data is sent to the destination. To build that state you need initial sync to run.

I am checking the doc, is it possible to manipulate the state stored in Airbyte DB?

Not using the API, only accessing the Airbyte DB directly