How do I make a incremental stream to do a full refresh? deleting the table and its corresponding _scd table and __airbyte__raw table, did not trigger a full data refresh

To make an incremental stream perform a full refresh in Airbyte, you can follow these steps: 1. Drop the final table: This will remove the existing data.

sql     DROP TABLE <your_final_table>;

2. Unset the

_airbyte_loaded_at

column: This will mark all records in the raw table as not loaded, forcing a reprocessing of all data.

sql     UPDATE airbyte_internal.<your_raw_table> SET _airbyte_loaded_at = NULL;

If you are using a nonstandard raw table schema, replace

airbyte_internal

with that schema. 3. Run a sync: After performing the above steps, trigger a sync in the Airbyte UI. This process will ensure that your final table is repopulated from the raw data, effectively performing a full refresh. For more detailed instructions, you can refer to the [Airbyte documentation on manually triggering a final table re-creation](https://docs.airbyte.com/using-airbyte/core-concepts/typing-deduping#manually-triggering-a-final-table-re-creation). If you need to schedule a full refresh periodically to reduce data inconsistencies over time, you might want to look into setting up a CRON job to reset the

streamState

to the

start_date

of the source. This way, the connection would pull all data over again since the

start_date

defined when you set up the source. This feature is discussed in [this GitHub issue](https://github.com/airbytehq/airbyte/issues/26513). If you have any further questions or need additional assistance, feel free to ask!

@kapa.ai both the solutions didn't work for me, is it possible to change connection state of a incremental strem, in perticular I'm thinking about changing cursor attribute to oldest date

Yes, it is possible to change the connection state of an incremental stream, including modifying the cursor attribute to an older date. However, this should be done with caution as it can potentially break your syncs and may require a full historical sync to fix. Here are the steps to change the connection state in the Airbyte UI: 1. In the Airbyte UI, click Connections and then click the connection you want to display. 2. Click the Settings tab on the Connection page. 3. Click the Advanced dropdown arrow. 4. Connection State will be displayed. 5. Click anywhere in the Connection state to start editing. 6. Modify the cursor attribute to the desired date. 7. Confirm changes by clicking "Update state". Discard any changes by clicking "Revert changes". 8. Confirm the changes to the connection state update. Important Note: Updates to the connection state should be handled with extreme care. Incorrect updates may break your syncs, requiring a full historical sync of your data to fix. Make changes only as directed by the Airbyte team. For more detailed information, you can refer to the [Airbyte documentation on managing connection state](https://docs.airbyte.com/cloud/managing-airbyte-cloud/manage-connection-state). If you prefer to update the state directly in the database, you can use the following SQL command, but be aware that this method is more advanced and should be used with caution:

sql UPDATE state SET state = jsonb_set(state, '{state, streams, 0, cursor}', '"NEW CURSOR"', false) WHERE connection_id = 'CONNECTION ID';

For example, if you want to set the cursor to the oldest date, replace

"NEW CURSOR"

with the desired date in the correct format. For more context, you can refer to the forum post where a user describes a similar process: [Forum Post: Update the cursor manually breaks my connection](https://discuss.airbyte.io/t/393). If you encounter any issues or need further assistance, it is recommended to reach out to the Airbyte support team or consult the community forums.