https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> Can you tell in one of my connectio...
# ask-ai
d
@kapa.ai Can you tell in one of my connections same data is getting incrementally appended post every sync what could be an issue looks like connection state is not updating the cursor date.
k
If the same data is being incrementally appended after every sync in your Airbyte connection, a common cause is that the cursor field (which tracks the last synced record) is not being updated correctly in the connection state. This can happen for several reasons: 1. Cursor Precision Issues: If your cursor field (e.g., a datetime column) is being rounded or losing precision, Airbyte may not recognize that it has already synced certain records. For example, with some connectors like MSSQL, there have been issues where the datetime cursor is rounded down, causing the same records to be re-synced because the state does not reflect the true last value with full precision. This results in repeated syncing of the same data on each run, as the cursor in the state is always less than the actual data's value with higher precision, or is not updated at all if the precision is lost [[source-mssql] Cursor datetime seems to be rounding the time on incremental append](https://github.com/airbytehq/airbyte/issues/48403), [Source MSSQL: datetime fields with millisecond precision are being rounded down to the second](https://github.com/airbytehq/airbyte/issues/10262). 2. Inclusive Cursor Logic: Airbyte uses an "at-least-once" delivery guarantee, and the incremental sync query typically uses 
cursor_field >= last_synced_value
. If the cursor value is not granular enough (e.g., only to the second or day), or if the state is not updated to the latest value, the same record(s) may be included in every sync. This is expected behavior if the cursor is not unique or granular, but if you see this with high-precision fields, it may indicate a bug or misconfiguration ([Incremental Sync - Append: Inclusive Cursors](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append#inclusive-cursors)). 3. Connector Bugs or State Handling Issues: There are known issues where the connector or the Airbyte platform does not properly update or recognize the state, especially with custom connectors or certain source/destination combinations. This can cause the sync to always start from the same point, re-sending the same data [[Bug - Source connector made with connector Builder do not recognize the last date modification cursor in the connection state "streamState" in incrementa sync](https://github.com/airbytehq/airbyte/issues/61567)]. What you can do: - Check the precision and type of your cursor field. If possible, use a field with high granularity (e.g., a timestamp with milliseconds or a monotonically increasing ID). - Review the logs to see what value is being stored in the state after each sync and compare it to your source data. - If you are using a connector with known issues (like MSSQL with datetime), check for updates or workarounds in the Airbyte community or GitHub issues. - Consider switching to a deduplication sync mode or using Change Data Capture (CDC) if your use case allows. If you provide more details about your source, destination, and cursor field, I can help you further diagnose the issue.
d
current version of MSSQL is : 4.1.20
@kapa.ai using MSSQL is : 4.1.20 connector source
k
A message from kapa.ai
If you are using the MSSQL source connector version 4.1.20 and are experiencing the issue where the same data is incrementally appended after every sync, it is likely related to how the cursor field (typically a datetime column) is being handled. There are known issues with the MSSQL connector and cursor precision, where the datetime value used as a cursor may be rounded or not stored with full precision. This can cause Airbyte to re-sync the same records repeatedly, as the state does not accurately reflect the last synced value. This issue has been reported in several versions of the MSSQL connector, where the cursor datetime is rounded, leading to duplicate data being sent in subsequent syncs [[source-mssql] Cursor datetime seems to be rounding the time on incremental append](https://github.com/airbytehq/airbyte/issues/48403), [Source MSSQL: datetime fields with millisecond precision are being rounded down to the second](https://github.com/airbytehq/airbyte/issues/10262). Additionally, Airbyte's incremental sync uses an "at-least-once" delivery guarantee and typically queries with 
cursor_field >= last_synced_value
. If the cursor is not updated correctly or lacks sufficient granularity, the same records may be included in every sync ([Incremental Sync - Append: Inclusive Cursors](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append#inclusive-cursors)). What you can do: - Check if your cursor field is a datetime with sufficient precision (ideally including milliseconds). - Review the Airbyte sync logs to see what value is being stored in the state after each sync and compare it to your source data. - Consider upgrading to a newer connector version, as there have been ongoing fixes related to cursor handling in later releases ([MSSQL Changelog](https://docs.airbyte.com/integrations/sources/mssql#changelog)). - If possible, use a more granular or unique cursor field. If the problem persists even after these checks, it is likely due to the known cursor precision issue in the connector version you are using.
d
2025-07-13 211600.333 This is the updated date for cursor @kapa.ai
k
A message from kapa.ai
If your cursor value is 
2025-07-13 21:16:00.333
and you are using the MSSQL source connector, there are known issues where Airbyte may round or truncate the datetime value, especially with millisecond precision. This can cause the same records to be re-synced in every incremental sync, because the state may store the cursor as 
2025-07-13 21:16:00
(losing the 
.333
milliseconds), or in a different format, leading to incorrect comparisons during the next sync. This issue has been reported in the Airbyte community and GitHub, where datetime fields with millisecond precision are being rounded down, causing rows to keep incrementally syncing when they shouldn't ([Source MSSQL: datetime fields with millisecond precision are being rounded down to the second](https://github.com/airbytehq/airbyte/issues/10262), [[source-mssql] Cursor datetime seems to be rounding the time on incremental append](https://github.com/airbytehq/airbyte/issues/48403)). As a result, Airbyte may not update the cursor in the state with the full precision, and the next sync will re-select records with the same datetime value. Summary: If your cursor is 
2025-07-13 21:16:00.333
and you see repeated data, it is likely due to this precision issue in the connector. The knowledge sources confirm this is a known limitation with the MSSQL connector and datetime cursor fields.
Open in Slack
PreviousNext
Powered by