Hi Teams. I am currently facing a problem with mis...
# ask-community-for-troubleshooting
s
Hi Teams. I am currently facing a problem with missing records in Salesforce syncing. I am currently working with a Salesforce Contact table. The synchronization method is Incremental Sync - Deduped History The Contact table is heavily updated and pulls in tens of thousands of records daily. So I found that some records are missing. 1. Synchronization started on 2023-05-28 210000 (State is {SystemModstamp”: “2023-05-27T211508.000+0000} at this time) 2. after synchronization, State is updated to {“SystemModstamp”: “2023-05-28T211008.000+0000"}. 3. next synchronization, it was found that a few records were missing, records for which SystemModstamp had the following values: {“SystemModstamp”: “2023-05-28T211008.000+0000"}.(I checked the raw table) ◦ 2023-05-28 211507 UTC ◦ 2023-05-28 211506 UTC I can see why it is missing because the value of SystemModstamp, which is indeed a curl field, is the value after that at step2. How can this be resolved? https://discuss.airbyte.io/t/missing-records-in-salesforce-incremental-synchronization/4577
k
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, I can provide some information on how Airbyte handles incremental synchronization. In Incremental Sync - Deduped History mode, Airbyte uses a cursor field (in your case,
SystemModstamp
) to identify new or updated records since the last sync. It's possible that some records with the same
SystemModstamp
value as the state might be missed during the sync process. One possible solution mentioned in the [Insightly source changelog](https://docs.airbyte.com/integrations/sources/insightly) is to make incremental syncs timestamp inclusive. This change was implemented in version 0.1.3 of the Insightly connector. Although this is not directly related to the Salesforce connector, it might be worth considering a similar approach for your use case. I recommend checking the [Airbyte documentation](https://docs.airbyte.com) and [forum discussions](https://discuss.airbyte.io) for more information on handling missing records in incremental synchronization.
s
@kapa.ai Are there any other people or issues facing similar problems?
k
A message from kapa.ai
Yes, there are other users who have faced similar problems with different connectors. For example, in a [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1667981132722529), a user faced an issue with fetching the schema for the Adjust connector. Another user in the same conversation mentioned facing a similar issue with the Zuora connector. They also shared a [forum thread](https://discuss.airbyte.io/t/zuora-source-failed-to-fetch-schema/2976/10) discussing the issue. In another [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1668440072155119), a user faced an issue with the self-hosted Airbyte instance due to two tables with the same name in different schemas. The support team acknowledged it as a bug with the normalization process. These examples show that users have faced similar issues with different connectors, and the Airbyte team is actively working on resolving them.
s
@kapa.ai I think the case you are proposing is incorrect. You are mistaken. So the problem is that in incremental synchronization, data prior to the time of the saved state is not being taken properly. I don’t know if that is a bug in the API fetching or if Salesforce is trying to update the data as slightly past the time even though it was fine at the time.
k
A message from kapa.ai
I apologize for any confusion. Based on the information provided, it seems you are facing an issue with incremental synchronization in Airbyte, specifically with the Salesforce connector. Incremental synchronization has some limitations, as mentioned in the [documentation](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append#known-limitations). One of the limitations is that if a record is inserted or updated on the source system without correctly updating its cursor field value, it may not be replicated to the destination database. However, there are techniques to automatically update a timestamp upon modification, as mentioned in the [tutorial](https://airbyte.com/tutorials/incremental-data-synchronization). Another limitation is that if a record is modified multiple times between synchronization iterations, only the state of the record at the time of synchronization will be replicated to the destination. Other intermediate states will not be recorded. To address these concerns, you can consider using a different incremental approach based on binary logs, Write-Ahead-Logs (WAL), or [Change Data Capture (CDC)](https://docs.airbyte.com/understanding-airbyte/cdc). If you still face issues with the Salesforce connector, I recommend reaching out to the Airbyte community on their [Slack](https://airbytehq.slack.com) or [forum](https://discuss.airbyte.io) for further assistance.
s
@kapa.ai Do you know anyone in Airbyte’s Slack community who is familiar with the Salesforce connector?
k
A message from kapa.ai
Yes, there have been several users and support team members discussing the Salesforce connector in the Airbyte Slack community. Here are some examples: 1. In this [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1672862712633379), a user asked for help with the Salesforce connector's authentication process, and another user provided a link to a [tutorial](https://airbyte.com/tutorials/replicate-salesforce-data-to-bigquery#step-3) that helped them. 2. In another [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1667484217044359), a user faced an issue with the number of records retrieved by the Salesforce connector, and the support team acknowledged the issue and assigned it to the appropriate team. 3. In this [Slack conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1674017984608439), a user asked about the difference between the Salesforce connector and the Salesforce Marketing Cloud connector, and the support team provided clarification. You can join the Airbyte Slack community [here](https://airbytehq.slack.com) and ask for help or share your experience with the Salesforce connector. The community members and support team are usually responsive and helpful.
s
@kapa.ai Please send a mentions from this thread to anyone you think might be involved.
k
A message from kapa.ai
I apologize, but as an AI language model, I am unable to directly mention or tag users in the Airbyte Slack community. However, I can provide you with some guidance on how to approach the community for help. You can join the Airbyte Slack community [here](https://slack.airbyte.com/) and post your question in the appropriate channel, such as
#help-connections-issues
for questions or issues related to connections. Make sure to provide a clear description of your issue with the Salesforce connector and any relevant details. The community members and support team are usually responsive and helpful. Additionally, you can search for previous conversations related to the Salesforce connector in the Slack channels to see if someone has already faced a similar issue and received a solution.
s
@Marcos Marx (Airbyte) Do you have any good solutions?