Hello, I would like your recommendation and advice...
# feedback-and-requests
b
Hello, I would like your recommendation and advice. We move your information to our customer with ELT tools such as Airbyte. The type of information we move to our customer is wide, and in many cases it changes over time, can be corrected, or there is no set schedule, because it is sales/purchase information. We have several flows created in Airbyte, a problem that arises and we want to avoid is when the synchronization is done, and there is missing information, how to get the missing information without having to reprocess all the information. Our flows are configured as incremental, but our cursors are dates like 2021-12-20 or some are date and time 2021-12-20 000000 but we gave us there is no order, the loading date to the source has no order and does not always reflect the amount of new data. Ex: We synchronized the information of 2021-12-20 but the information of one location is missing, they upload the information and the cursor is the same, how to get only that information if the change is not reflected in the cursor. What options do we have on our side? I thought about modifying the cursor directly from the database and extracting but I want to hear your comments on this. Thanks.
Are you using a database to database connector? One way to have a better integration for this case is use Change Data Capture feature.
No, we connect to the database through a username and password through a server via SSH where we have access to those databases. What would be the most suitable database to database connector options?
The question is to understand if you're transfer data from Postgres to Mysql/Bigquery/Snowflake
I transfer from Oracle DB to Redshift.
r
Hi @Boggdan Barrientos what could be ideal for your use case is to have an
updated_at
field in your table that can be used as a timestamp and stores when a row was updated, this will allow all incremental loads to get the edited data. As Marcos suggested CDC could also help you for this use case but I'm afraid that we're not supporting CDC on Oracle DB.
Thanks @[DEPRECATED] Augustin Lafanechere, actually we have set a cursor that is a timestamp, but is not working as we need or as expected. Our client has many locations and their data is not upload in sequentially order. In many cases they upload a more freshness data before, and then past data. So when we compare the source data and our sync data, we have missed registries. Any idea if CDC is going to be supported for Oracle DB?