@kapa.ai Is it expected that the cursor for an incremental sync with the BigQuery source connector is using second-level precision? We have issues where we see that the cursor value sent in the query seems to be rounded or truncated to whole seconds, causing data to be synced twice. We are using a cursor field of type TIMESTAMP in the BigQuery source table.

The knowledge sources confirm how incremental sync with the BigQuery source works, but they do not explicitly mention any rounding/truncation of

TIMESTAMP

cursor values to second-level precision, nor any known bug about this behavior. So I can’t definitively say it is expected behavior. What is documented: - The BigQuery source uses a cursor filter of the form:

sql   WHERE cursor_field > last_cursor_value

for incremental syncs, and it supports

TIMESTAMP

as a recommended cursor type. [[BigQuery source](https://docs.airbyte.com/integrations/sources/bigquery#supported-sync-modes)] - Airbyte’s general incremental model is “at-least-once delivery,” and sources are allowed (and even encouraged) to re-send some data when cursors are not very granular or are ambiguous. This can lead to the same rows being emitted in multiple syncs, with deduplication handled on the destination side (especially with Incremental Append + Deduped). [[Inclusive cursors](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append-deduped#inclusive-cursors); [Incremental append](https://docs.airbyte.com/platform/using-airbyte/core-concepts/sync-modes/incremental-append#inclusive-cursors)] From the docs alone, we can say: - Duplicate syncing of some records is expected in principle due to the at-least-once guarantee. - However, the docs do not state that

TIMESTAMP

cursors are reduced to second precision; they only say that

TIMESTAMP

is supported and recommended. [[BigQuery source](https://docs.airbyte.com/integrations/sources/bigquery#supported-sync-modes)] Given that, if you are observing truncation/rounding to whole seconds on a

TIMESTAMP

cursor, that behavior is not clearly documented in the available sources, and I cannot confirm it is intentional or expected. If this second-level truncation is causing more duplicates than you’d expect from normal at-least-once semantics, the next step would be to: - Verify the exact

last_cursor_value

stored in the connection state in Airbyte’s UI or logs. - Compare it to the value used in the generated BigQuery query (if visible in logs) to see where precision is lost. But based on the provided documentation, I can only say that duplicates are generally expected, while the specific second-level precision behavior is not described.

@kapa.ai I have verified that the last cursor value sent in the BigQuery query is on a second level. This means that anything that was already synced previously that has a timestamp on the same second value will be synced again in the next sync.

A message from kapa.ai