Hey team, I have a scenario where I'd like to do an incremental pull from some very large tables so I don't have to fully refresh data that's not changing. I believe there is a useful modified date on those tables, so that's good for the incremental, but there's often not a decent reliable unique key. Is there any way to manage this in Airbyte? My hack thought is to have a process that deletes the prior 30 days of data (based on a transaction date in the data) from the target table and then let Airbyte do an incremental refresh using that date as the modified date, which would theoretically replace any transactions that were just deleted. I haven't tested this, and I am not sure how Airbyte handles the high water mark (query at runtime vs. stored metadata), so I don't know if this would even work. Any thoughts for this kind of "sliding window full replace" activity (or just a generally better way to do this)?

Hello Brian Nelson, it's been a while without an update from us. Are you still having problems or did you find a solution?

@Marcos Marx (Airbyte) Never found a solution, never tested my hack theory.