Hello ! I’m learning airbyte and played with some connectors to be prepared to develop my own. I got a question: how does incremental sync know when the reading from source connector finishes ? From my understanding, if there are many records with the same cursor value, the state emitted from the read will be same for the reads. So how does airbyte sync worker know it should not continue reading from the source connector ? Thanks a lot !

It doesn't, the conector just finishes it's loop and so the container dies. Airbyte literally just run a container with the read command and passing some arguments. All transfers to the destination connector is done via stdout https://airbytehq.github.io/understanding-airbyte/airbyte-protocol-docker

Are you saying there is always only one read being called to the source connector ? And the connector will start streaming all it has to the STDIN ? Even in the incremental mode ?

Any airbyte sync will output it's source data to the destination via stdout/stdin (depending on the point of view). So, yes, even incremental will do that. Including states, they are sent to the destination also, because airbyte needs to check if the data that was sent actually was written. Yes, the checkpointing is only to control when state messages will be emitted (not saved). The worker knows that there is no more data because the container exits with a exit code of 0 (or failed if more than that, Unix style)

@Marcos Marx (Airbyte) turned this thread into Zendesk ticket 3058 to ensure timely resolution!

I'm not sure if they go into this much detail but I found these tutorials helpful in understanding sync flow: https://airbyte.com/tutorials/incremental-data-synchronization