Are there specific reasons that an S3 Source conne...
# replication-ask-ai
j
Are there specific reasons that an S3 Source connector would only return 75% of the records from a parquet file in S3, missing the other records?
k
Hmm, I don't know enough to give you a confident answer yet. The extracts provided do not contain specific information about an issue where an S3 Source connector would only return 75% of the records from a parquet file in S3. However, it's worth noting that there can be various reasons for such behavior, including but not limited to, issues with the file itself, configuration issues, or even rate limiting. For more detailed assistance, I would recommend checking out the [Airbyte documentation](https://docs.airbyte.com/integrations/sources/file) or posting your question on the [Airbyte forum](https://discuss.airbyte.io/).
j
@kapa.ai The actual connection sync does not show any errors. It runs successfully but is failing to capture all the records
@kapa.ai Could adjusting record batch size or buffer size parameters fix this?
k
There could be a few reasons why your connection sync is not capturing all the records. 1. If you're using Incremental sync mode, it's important to note that this mode relies on a cursor field to determine which records to sync. If modifications to the underlying records are made without properly updating the cursor field, then the updated records won't be picked up by the sync as expected. The source connectors extract delta rows using a SQL query like this:
Copy code
sql SELECT * FROM table WHERE cursor_field >= 'last_sync_max_cursor_field_value'
(Source: [Incremental Sync - Append](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append/)) 2. If you're using Incremental - Append + Deduped sync mode, the same limitation applies. The SQL query used in this case is:
Copy code
sql select * from table where cursor_field > 'last_sync_max_cursor_field_value'
(Source: [Incremental Sync - Append + Deduped](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append-deduped)) 3. There could be a limitation or issue with the specific connector you're using. For example, a user reported an issue where the Salesforce connector retrieved only 14k records instead of 49k+ records. The support team suggested creating a GitHub issue so the team can look into it. (Source: [Forum Post](https://discuss.airbyte.io/t/3094)) If none of these scenarios apply to your situation, it would be helpful to have more information about your specific setup and the issue you're experiencing. You might consider reaching out to the Airbyte community on their [forum](https://discuss.airbyte.io/) or [Slack](https://airbytehq.slack.com) for further assistance.
A message from kapa.ai
Based on the information provided in the extracts, there is no mention of adjusting record batch size or buffer size parameters as a solution to the issue of a connection sync not capturing all records. The extracts primarily discuss the limitations of the Incremental sync mode, where if modifications to the underlying records are made without properly updating the cursor field, then the updated records won't be picked up by the sync as expected. If you're experiencing issues with your sync not capturing all records, it's recommended to review your sync history and logs for any potential issues. You can do this by navigating to the Connections section on the [Airbyte Cloud](http://cloud.airbyte.com/) dashboard and clicking on a connection to view its sync history and logs ([source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/review-sync-history)). If the issue persists, consider reaching out to the Airbyte community on their [forum](https://discuss.airbyte.io/) or [Slack](https://airbytehq.slack.com) for further assistance.