@kapa.ai i have a sync that extracts data from a tablet in Netsuite to a table in snowflake. This stream is set as incremental + deduplicate. Loading the full history works, but if i trigger the sink after the initial load it does not load new records even though there are new records in netsuite. In order words, the data in snowflake is never getting updated. What can the cause?

The issue you're experiencing with the incremental + deduplicate sync mode not loading new records into Snowflake, despite new records being present in NetSuite, could be related to how the cursor field is being managed. Here are some potential causes and steps to troubleshoot: 1. Cursor Field Not Updated Properly: - The incremental sync relies on a cursor field to track changes. If modifications to the underlying records are made without properly updating the cursor field, the updated records won't be picked up by the incremental sync. This is because the source connectors extract delta rows using a SQL query that looks like:

sql      select * from table where cursor_field > 'last_sync_max_cursor_field_value'

- Ensure that the cursor field in your NetSuite table is being updated correctly whenever new records are added or existing records are modified. [Source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#known-limitations). 2. No Modifications Detected: - If a sync is executed without making any modifications on the source database, no records will be emitted, and no changes will be made on the destination database. Verify that there are indeed new or modified records in NetSuite that should be picked up by the sync. [Source](https://airbyte.com/tutorials/incremental-data-synchronization#step-7). 3. Type Errors or Schema Validation Issues: - There could be type errors or schema validation issues that are preventing the incremental sync from processing new records. Check the logs for any errors or warnings that might indicate such issues. For example, JSON schema validation failures or issues with the binlog file positions could be relevant. [Source](https://github.com/airbytehq/airbyte/issues/17512). 4. Normalization Settings: - Ensure that your normalization settings are correctly configured. If you have normalization enabled, make sure it is set up to handle the data types and structures in your NetSuite table. [Source](https://discuss.airbyte.io/t/2740). To further diagnose the issue, you can: - Review the sync logs for any errors or warnings. - Verify that the cursor field in NetSuite is being updated correctly. - Ensure that there are indeed new or modified records in NetSuite that should be picked up by the sync. - Check your normalization settings and make sure they are correctly configured. If you continue to experience issues, you may want to reach out to the Airbyte community or support for further assistance.