Hi everyone, following connection is used: Stripe ...
# help-connector-development
f
Hi everyone, following connection is used: Stripe 3.4.1 -> Snowflake 0.4.63 I'm using Incremental/Append Mode for most of the streams. The cursor is based on "created". Problem: With every Sync (once per Day), records will be written in the table, which have been written with the sync the day before. So e.g. if a promotion code is emitted by the sync on day 1, it will also be emitted on day 2, even if there was no change at all on this entity. Does anyone have a solution on this behaviour, so that the sync does not add already existing records to the table?
k
A message from kapa.ai
It seems like you are experiencing a behavior where the same row is being emitted during each sync, even if there is no change in the data. This can occur in Incremental Append mode when the cursor field is not very granular or when the source is run multiple times within the same cursor value range. By convention, sources should prefer resending data if the cursor field is ambiguous [source](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append#inclusive-cursors). To avoid this issue, you can try the following: 1. Make sure your cursor field has a higher granularity, such as including hours, minutes, and seconds, instead of just the date. This will help in reducing the chances of ambiguity in the cursor field. 2. If possible, adjust the sync frequency to ensure that the source is not run multiple times within the same cursor value range. Remember that in Incremental Append mode, Airbyte provides an "at least once" guarantee of replicating each record that is present when the sync runs [source](https://docs.airbyte.com/understanding-airbyte/connections/incremental-append#overview). If you still face issues, consider reaching out to the Airbyte community on their [forum](https://discuss.airbyte.io) or [Slack](https://airbytehq.slack.com) for further assistance.
a
You can always deduplicate your raw data from airbyte using
qualify
and
partition by
SQL keywords like this :
Copy code
qualify row_number() over(partition by id_promotion order by created_at asc) = 1