Hi all, I'm trying to get deeper into the Incremental Append streams. Sometimes I see a lot of messages like these in case of first sync logs: 2022-12-21 102706 source > State report for stream campaigns_email_content - original: null = null (count 0) -> latest: id_email = 17022278 (count 1) I suppose it means creation of "incremental cursor" of some kind, am I right? The strange thing is - in some syncs there is a lot of them, in another syncs there are almost none, just the "data pump" messages: ... 2022-12-21 092103 destination > Flushing email_content: 65 records (24 MB) 2022-12-21 092103 source > Reading stream email_content. Records read: 750000 2022-12-21 092103 INFO i.a.w.g.DefaultReplicationWorker(lambda$readFromSrcAndWriteToDstRunnable$6):347 - Records read: 750000 (55 GB) 2022-12-21 092103 destination > Flushing email_content: 94 records (24 MB) 2022-12-21 092104 destination > Flushing email_content: 96 records (24 MB) ... My questions are - is it an indication of cursor creating or what? Why does it take so much time and can I add resource to make it go faster? And what happens if I reset the data, is the cursor deleted as well?

Hi Svatopluk! Thanks for your patience - I was out last week for the holidays and am catching up with messages now. I'm not quite sure what that field refers to, could you include the full logs? What connectors are you using? Making syncs faster is something that we're working on. I don't believe this can be sped up, but please take a look at this doc to see what can be given more resources: https://docs.airbyte.com/operator-guides/scaling-airbyte I believe that if you've got a user defined cursor field, that would be deleted if you reset the data. Some connectors have automatically defined cursor fields and in that instance they would not be affected.

Hi Nataly, thanks for the reply, I've overlooked that. We've set up some environment where the load speed is quite acceptable for now, so not an issue anymore. (Still there is a room for improvement, good to hear you are working on it 🙂 ) Have a nice day.