https://linen.dev logo
Join Slack
Powered by
I have the error ```Failure in source: Saved offse...
# ask-ai
s
I have the error
Copy code
Failure in source: Saved offset no longer present on the server. Please reset the connection, and then increase binlog retention and/or increase sync frequency. See <https://docs.airbyte.com/integrations/sources/mysql/mysql-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs> for more details.
how can I trigger a full refresh in Airbyte 0.63.1?
k
To trigger a full refresh in Airbyte, you need to reset the data in your connection. Here are the steps to do so: 1. Select Connections in the main navbar. 2. Select the connection(s) affected by the update. 3. Select the Replication tab. 4. Select Refresh source schema. 5. Select OK. Any detected schema changes will be listed for your review. 6. Select Save changes at the bottom of the page. 7. Ensure the Reset all streams option is checked. 8. Select Save connection. This will reset the data in your destination and initiate a fresh sync. For more information on resetting your data in Airbyte, see [this page](https://docs.airbyte.com/operator-guides/clear). Please note that the error message you're seeing is due to the binlogs needed for the incremental sync being removed by MySQL. This can occur when there are lots of database updates resulting in more WAL files than allowed in the 
pg_wal
directory. To prevent this, you can sync the data source more frequently or set a higher 
binlog_expire_logs_seconds
. It's recommended to set this value to a time period of 7 days. The downside of this approach is that more disk space will be needed. You can find more details [here](https://docs.airbyte.com/integrations/sources/mysql/mysql-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs). Sources: - [Monday Migration Guide](https://docs.airbyte.com/integrations/sources/monday-migrations#refresh-schemas-and-reset-data) - [MySQL Troubleshooting](https://docs.airbyte.com/integrations/sources/mysql/mysql-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs)
s
@kapa.ai Reset all streams is no longer available, only "clear" and "refresh"
k
Yes, you're correct. As of Airbyte v1.0.0, the term "Clear" has replaced "Reset" in the Airbyte UI and documentation. The "Clear" functionality allows you to erase all of the data that Airbyte has created in your destination. This can be accomplished with a Clear sync - either clearing a single stream, or all streams. Please note that there is no way to recover from a clear sync, so be certain that you wish to erase all the data in your destination. A Clear can be triggered either from the UI or Airbyte API. Airbyte allows you to clear all streams in the connection or only a single stream through the UI. You may also be prompted to clear some streams when making configuration changes that apply to multiple streams. Airbyte additionally supports the clearing of multiple streams through the API [[source](https://docs.airbyte.com/operator-guides/clear)]. In addition to "Clear", Airbyte has introduced "Refresh" syncs. A Refresh Sync erases the current cursor you have set for any incremental streams, effectively ‘rewinding’ the sync back to the beginning. This is helpful in scenarios where the source you're syncing from does not sync record deletions/removals, the source you are syncing from is unreliable and you wish to re-attempt the sync, or something has gone wrong with your sync, and you wish to try again. Refresh Syncs do not create data downtime. While the Refresh Sync is running, the previous data in your destination remains available to query. Only at the successful conclusion of a Refresh Sync will the newly refreshed data be swapped or merged into place [[source](https://docs.airbyte.com/operator-guides/refreshes#refreshes-overview)]. So, in summary, "Reset" has been replaced with "Clear" and "Refresh" functionalities in Airbyte.
s
@kapa.ai how can I trigger a CDC-based full refresh after I "clear" the stream?
k
A message from kapa.ai
To trigger a CDC-based full refresh after you "clear" the stream, you would need to follow these steps: 1. Clear the stream: Navigate to the connection settings in the Airbyte UI, select the stream you want to clear, and choose the "Clear" option. This will erase all the data that Airbyte has created in your destination for that particular stream. Please note that this action is irreversible [[source](https://docs.airbyte.com/operator-guides/clear)]. 2. Set up CDC: If you haven't already, you'll need to set up Change Data Capture (CDC) for your source. This process varies depending on the source type. For example, for a Postgres source, you would need to enable the CDC mode. For a MongoDB source, you would need to utilize the change streams feature of a MongoDB replica set [[source](https://docs.airbyte.com/integrations/sources/postgres/postgres-troubleshooting#under-cdc-incremental-mode-there-are-still-full-refresh-syncs), [source](https://docs.airbyte.com/integrations/sources/mongodb-v2#replication-methods)]. 3. Trigger a full refresh: After clearing the stream and setting up CDC, you can trigger a full refresh. This can be done by selecting the "Full Refresh" option in the connection settings in the Airbyte UI. This will read the entire contents of the source and load it into your destination [[source](https://docs.airbyte.com/connector-development/cdk-python/full-refresh-stream)]. Please note that the exact steps may vary depending on your specific source and destination types. Always refer to the specific documentation for your source and destination for the most accurate information.
s
@kapa.ai i no longer see this option
This can be done by selecting the "Full Refresh" option in the connection settings in the Airbyte UI.
k
A message from kapa.ai
I apologize for the confusion. The "Full Refresh" option might not be available for all sources or under certain conditions. For example, in the case of the Postgres source, a user reported that the "Full Refresh" option was not available in the dropdown menu of sync mode after an upgrade to a newer version of Airbyte [[source](https://github.com/airbytehq/airbyte/issues/28280)]. If you're not seeing the "Full Refresh" option, it could be due to the specific source connector you're using or the version of Airbyte you're running. It's recommended to check the documentation for your specific source connector to understand the supported sync modes. For example, the Freshdesk source connector supports Full Refresh - Overwrite, Full Refresh - Append, Incremental - Append, and Incremental - Append + Deduped [[source](https://docs.airbyte.com/integrations/sources/freshdesk#supported-sync-modes)]. If you're still having trouble, consider reaching out to the Airbyte community or support for further assistance.
A message from kapa.ai
I apologize for the confusion. The "Refresh source schema" option is not the same as the "Full Refresh" sync mode. The "Refresh source schema" option is used to manually update the source schema to stay up to date with changes in your schema. Here are the steps to manually refresh the source schema: 1. In the Airbyte UI, click Connections and then click the connection you want to refresh. Click the Schema tab. 2. In the Select streams table, click Refresh source schema to fetch the schema of your data source. 3. If there are changes to the schema, you can review them in the Refreshed source schema dialog [[source](https://docs.airbyte.com/using-airbyte/schema-change-management#manually-refresh-the-source-schema), [source](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-schema-changes#manually-refresh-the-source-schema)]. If you're not seeing the "Refresh source schema" option, it could be due to the specific source connector you're using or the version of Airbyte you're running. It's recommended to check the documentation for your specific source connector or reach out to the Airbyte community or support for further assistance.
28 Views
Open in Slack
PreviousNext
Powered by