We have a full refresh overwrite table within a sy...
# replication-ask-ai
j
We have a full refresh overwrite table within a sync the stream team and team_membership from github, that runs every day. The ingestion sometimes misses some records that exist in the source. Also we re-sync missed data sometimes save all data, and sometime misses records again. What could be causing that?
k
Hmm, I don't know enough to give you a confident answer yet. The extracts provided do not contain specific information about the issue you're experiencing with the full refresh overwrite table within a sync for the stream team and team_membership from Github. However, based on the information available, it's important to note that in the Full Refresh - Overwrite mode, Airbyte retrieves all available information requested from the source, regardless of whether it has been synced before. New syncs will destroy all data in the existing destination table and then pull the new data in. Therefore, data that has been removed from the source after an old sync will be deleted in the destination table [source](https://docs.airbyte.com/understanding-airbyte/connections/full-refresh-overwrite). If you're missing some records, it could be due to a variety of factors not covered in the extracts, such as network issues, changes in the source data between syncs, or issues with the specific connector you're using. For more detailed help, I would recommend checking out the [Airbyte Documentation](https://docs.airbyte.com/), or asking for support on the [Airbyte Forum](https://discuss.airbyte.io) or [Slack](https://airbytehq.slack.com).
j
We are using the github source connector, and as destination, faros destination @kapa.ai
k
Based on the information provided in the extracts, here's how you can set up a connection from GitHub to Faros using Airbyte: 1. In the Airbyte web app, click on the new source button in the top right corner of the app to go to the page to add a new Airbyte source. 2. Enter the name github-source as the source name and select Github connector as source type. 3. After selecting the GitHub source type, you will be presented with two text boxes. The first is to enter a repository you want. In this box, type in airbytehq/airbyte. In the second box, you will need to provide a GitHub access token which you can obtain from [here](https://github.com/settings/tokens). 4. Make sure you grant the token the repo and write:discussion permissions. After you've filled all fields, hit the set up source button. 5. If the setup was successful, you will be taken to the destination screen where you will add a new destination. 6. Click the add destination button, and, in the drop down that follows, click add new destination. Then, you will see a page to add the destination name. Type in the name we gave the Faros destination. 7. After, you will be presented with some text boxes to enter the database connection details. Enter the values for the Faros destination. 8. Then click on the Set up destination button. If your credentials are all good for the database, the Faros destination would have been set, and now you will need to make the connection from the source (GitHub) to the destination (Faros). Please note that the specific details for setting up a Faros destination are not provided in the extracts. You may need to refer to the Faros documentation or support for specific instructions on setting up a Faros destination. Source: [Build a GitHub activity dashboard for your project](https://airbyte.com/tutorials/build-a-github-activity-dashboard-for-your-project)
j
We have a full refresh overwrite table within a sync the stream team and team_membership from github, that runs every day. The ingestion sometimes misses some records that exist in the source. Also we re-sync missed data sometimes save all data, and sometime misses records again. We are using the github source connector, and as destination, faros destination. What could be causing that? @kapa.ai
k
A message from kapa.ai
Based on the information provided in the extracts, there could be several reasons why some records are missed during the sync process. Here are a few possibilities: 1. API Limitations: The GitHub source connector should not run into GitHub API limitations under normal usage. However, if you're making a large number of requests in a short period of time, you might hit the rate limit. If you see any rate limit issues that are not automatically retried successfully, you should [create an issue](https://github.com/airbytehq/airbyte/issues) on GitHub. 2. Sync Modes: The GitHub source connector supports several [sync modes](https://docs.airbyte.com/cloud/core-concepts/#connection-sync-modes), including Full Refresh - Overwrite, Full Refresh - Append, Incremental Sync - Append, and Incremental Sync - Append + Deduped. If you're using the Full Refresh - Overwrite mode, it's possible that some records are being overwritten during the sync process. 3. Connector Issues: There might be issues with the GitHub source connector or the Faros destination connector. If you suspect this is the case, you can check for common troubleshooting issues for the GitHub connector on the [Airbyte Forum](https://github.com/airbytehq/airbyte/discussions). If you can't find a solution there, you can [open a GitHub issue](https://github.com/airbytehq/airbyte). 4. Data Issues: There might be issues with the data itself. For example, if a record is deleted in the source after the sync process starts but before it finishes, that record might not be included in the destination. Without more specific information, it's hard to say exactly what might be causing the issue. I would recommend starting with these suggestions and seeing if any of them help. If not, you might want to reach out to Airbyte support for more assistance.