Hi All, I have been using Airbyte to sync data fro...
# replication-troubleshootings
Satya Varma
10/12/2022, 6:12 AMHi All, I have been using Airbyte to sync data from Salesforce to redshift. We are facing an issue of missing data when there is a huge load. And we found a way to backfill the data by updating the status in state table. But, don’t have an idea of how to pass end time also to backfill the data only in a required time range rather than backfilling the entire data from a starting point. If any one have any idea/lead, Please let me know.
Thanks in advance
✍️ 1
u
user
10/12/2022, 11:57 AM@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2751 to ensure timely resolution!
e
Eli Sigal
10/12/2022, 1:33 PMHi.
Not sure if I understood correctly the issue here-
You are polling SF data to redshift and there is data missing and the only way to get it - is to run the job again with different time range for SF?
Eli Sigal
10/12/2022, 1:34 PMIf this is the case- an issue needs to be opened since you can batch get data from SF without polling time ranges manually - we do it though not using airbyte but rather a service we've created.
so a pr could be possible here if this is the case.
if not- please elaborate 🙂
s
Satya Varma
10/13/2022, 6:44 AM@Eli Sigal yup, that’s the issue.
m
Marcos Marx (Airbyte)
10/14/2022, 5:39 PMYou can update the state @Satya Varma which will change the
start_dateend_dates
Satya Varma
10/18/2022, 5:43 AM@Eli Sigal @Marcos Marx (Airbyte) Is there a way to define airbyte hash key column values. So that, we can write our own job for backfilling the data.
m
Marcos Marx (Airbyte)
10/18/2022, 11:49 AMNo, @Satya Varma today the hash key column is generated using uuid in the destinatinon. Of course you can customize the destination to achieve what you want but this will loose parity with official connector