Hello, I would like your recommendation and advice. We move your information to our customer with ELT tools such as Airbyte.
The type of information we move to our customer is wide, and in many cases it changes over time, can be corrected, or there is no set schedule, because it is sales/purchase information.
We have several flows created in Airbyte, a problem that arises and we want to avoid is when the synchronization is done, and there is missing information, how to get the missing information without having to reprocess all the information.
Our flows are configured as incremental, but our cursors are dates like 2021-12-20 or some are date and time 2021-12-20 000000 but we gave us there is no order, the loading date to the source has no order and does not always reflect the amount of new data. Ex: We synchronized the information of 2021-12-20 but the information of one location is missing, they upload the information and the cursor is the same, how to get only that information if the change is not reflected in the cursor.
What options do we have on our side? I thought about modifying the cursor directly from the database and extracting but I want to hear your comments on this. Thanks.