Hi. Does anyone have problems with the Airbyte Postgres connector? I have noumerous streams syncing, some as full-refresh/overwrite and some as incremental append. However, the full-frefresh overwrite streams seems to also only fetch new records since the last sync while also overwriting. So these streams are left with only the new records. I have reset the data, and then it emitts all data, but then the next sync fails agian. Thanks

It sounds like you are experiencing issues with the full refresh overwrite mode in the Airbyte Postgres connector. Based on the information from the knowledge sources, here are a few points and potential steps to address the problem: 1. Full Refresh Overwrite Behavior: - In full refresh overwrite mode, Airbyte is expected to extract and emit all records from the source data set during each sync, regardless of whether they have been previously replicated to the destination or not. This means that every record should be overwritten in the destination during each sync. - You can verify this behavior by checking the

_airbyte_emitted_at

and

_airbyte_normalized_at

timestamps, which should confirm that every record has been overwritten and re-normalized. [Source](https://airbyte.com/tutorials/full-data-synchronization#step-7). 2. Common Issues and Troubleshooting: - Logs and Errors: Ensure to check the logs for any errors or messages that might indicate why the sync is not behaving as expected. Sometimes, the logs might not show explicit errors but can provide hints about what might be going wrong. - Data Reset: As you mentioned, resetting the data causes all records to be emitted correctly. However, subsequent syncs fail. This could indicate an issue with how the connector handles state or incremental updates. - Version Compatibility: Ensure that you are using compatible versions of Airbyte, the Postgres source connector, and the destination connector. Sometimes, version mismatches can cause unexpected behavior. 3. Known Issues and Community Reports: - There have been reports of similar issues where full refresh overwrite syncs do not behave as expected. For example, users have reported that the first sync saves all data, but subsequent syncs delete all data on the destination and do not insert anything, and then the next sync inserts all data again. [Source](https://github.com/airbytehq/airbyte/issues/2607). - Another user reported that some streams do not sync properly, especially small tables, and the issue seems to happen intermittently. [Source](https://discuss.airbyte.io/t/2049). 4. Actionable Steps: - Isolate the Issue: Create separate connections for the problematic streams and observe their behavior. This can help isolate whether the issue is specific to certain streams or a general problem with the connector. - Open an Issue: If the problem persists, consider opening an issue on the Airbyte GitHub repository with detailed information about your setup, logs, and any steps you have already taken. This can help the Airbyte team and community provide more targeted assistance. [Example Issue](https://github.com/airbytehq/airbyte/issues/12771). By following these steps, you should be able to diagnose and potentially resolve the issues with your Airbyte Postgres connector. If you need further assistance, feel free to reach out to the Airbyte community or support channels.