I’m implementing Incremental sync for one of my st...
# ask-ai
a
I’m implementing Incremental sync for one of my streams so that not all of the data needs to be fetched, but just those records that have been updated. But it seems airbyte doesn’t support a timestamp value to rely on. This is what my API says in the docs (The timestamp value returned has no relation with actual date or time. As such it cannot be converted to a date\time value. The timestamp is a rowversion value). Is it possible to use this value?
u
Based on the information provided in the knowledge sources, it appears that Airbyte typically relies on datetime-based cursor fields for incremental syncs. However, your API's timestamp value doesn't correspond to an actual date or time, which presents a challenge. While Airbyte doesn't have built-in support for non-datetime cursor values, there might be a workaround you can use. The Delighted connector example mentioned in one of the forum posts provides a potential solution: [Forum Post: Handling UNIX timestamps with Low-Code CDK](https://discuss.airbyte.io/t/3921) suggests using a custom configuration for the DeclarativeStream to handle non-standard timestamp formats. Although this example is for UNIX timestamps, you might be able to adapt it for your use case. Here's a potential approach you could try: 1. Use the rowversion value as your cursor field. 2. In your connector configuration, set up the incremental sync to treat this value as a string or integer, rather than a datetime. 3. Modify the comparison logic to use greater than/less than operations on this value, rather than datetime comparisons. However, it's important to note that this is not a standard use case, and you may need to implement a custom solution or extend the existing Airbyte functionality to fully support this type of incremental sync. If you're developing a custom connector, you might need to implement your own logic for handling this type of cursor value in your
read_records
method. Keep in mind that without seeing the exact structure of your API response and the full capabilities of your API, it's difficult to provide a more specific solution. You might want to consult with Airbyte's community or support channels for more tailored advice on implementing this non-standard incremental sync approach.