I have a question. Why does Airbyte raw data have no unique identifier by stream slices?

@Zaza Javakhishvili I didn’t understand your question, can you give more context or explain your use case?

@Marcos Marx (Airbyte) Ok, So some source connector streams does not support incremental update right? You have two option in this case: 1. full refresh - overwrite 2. full refresh - append If you do overwrite, you have no history. But if you need to have history, you need to append. In case of append you have no any unique identifier to detect stream run/sync iteration. there are columns like: _airbyte_ab_id - is not unique per run/sync iteration _airbyte_emitted_at - it is not unique per run/sync iteration _airbyte_data - possible not contain date values to use. So you need to do extra steps and compare raw and find max _airbyte_emitted_at dates... It must have unique ID witch will show stream run/sync iteration.

Hi. Yes , you are correct. you need to do extra work to get a merge logic.

@Eli Sigal So why not to add the sync iteration unique identifier?

@Zaza Javakhishvili your issue is a duplicate of https://github.com/airbytehq/airbyte/issues/4025 this feature request will not get prioritize for next quarter unfortunately. Feel free to add more context to this issue or explain your use case to help implementing the feature in the future.