"date_end_session": 1727219414,

im using this field as incremental sync, what should my start datetime format should be @kapa.ai

The

date_end_session

field you're using for incremental sync appears to be in Unix timestamp format, which is a common way of storing date and time information. Unix timestamps represent the number of seconds that have passed since 000000 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds. However, based on the knowledge sources provided, Airbyte's incremental sync configuration typically uses datetime formats like

%Y-%m-%dT%H:%M:%S.%fZ

,

%Y-%m-%dT%H:%M:%S.%f+00:00

, or

%Y-%m-%dT%H:%M:%S+00:00

. These formats are based on the ISO 8601 datetime standard, not Unix timestamps. If you want to use a Unix timestamp for incremental sync in Airbyte, you might need to convert it to a compatible datetime format first. Unfortunately, the provided knowledge sources do not include information on how to do this directly within Airbyte. For more details on configuring incremental syncs in Airbyte, you can refer to the [official documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration).

can i use %s with 1727244577 as startdate time @kapa.ai

A message from kapa.ai

Yes, you can use

%s

as the datetime format for Unix timestamps in Airbyte. The

%s

placeholder represents the Epoch Unix timestamp, which is the number of seconds that have passed since 000000 Thursday, 1 January 1970, Coordinated Universal Time (UTC), minus leap seconds. Here's an example of how you might configure this in your connector:

yaml incremental_sync:   type: DatetimeBasedCursor   cursor_field: "date_end_session"   datetime_format: "%s"   start_datetime: "1727244577"

In this configuration,

date_end_session

is the cursor field,

%s

is the datetime format for Unix timestamps, and

1727244577

is the start datetime. Please note that this is a simplified example and your actual configuration might need additional parameters depending on your specific use case. For more information, you can refer to the [official Airbyte documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor) on the

DatetimeBasedCursor

object.

what is the cursor granularity for 1727244577 @kapa.ai

A message from kapa.ai

The cursor granularity for a Unix timestamp like

1727244577

would be

PT1S

, which represents a time duration of one second in ISO 8601 format. This is because Unix timestamps have a granularity of one second. Here's how you might specify this in your Airbyte configuration:

yaml incremental_sync:   type: DatetimeBasedCursor   cursor_field: "date_end_session"   datetime_format: "%s"   start_datetime: "1727244577"   cursor_granularity: "PT1S"

In this configuration,

PT1S

is the cursor granularity, indicating that the smallest increment for the datetime format is one second. For more information, you can refer to the [official Airbyte documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor) on the

DatetimeBasedCursor

object.

is incremental posssible in EPOCH timestamp, the sync doesn't seem to be incremental, getting same sets of record for all syncs @kapa.ai

A message from kapa.ai

Yes, incremental sync is possible with EPOCH timestamps. However, there are a few things to consider if you're seeing the same sets of records for all syncs: 1. Cursor Field: Ensure that the cursor field (in your case, the EPOCH timestamp field) is being updated in the source data. If the cursor field isn't changing, Airbyte will keep pulling the same data. 2. At-Least-Once Delivery: Airbyte provides an at-least-once delivery guarantee. This means that it's acceptable for sources to re-send some data when ran incrementally. If your data hasn't changed and you attempt to run additional syncs, the cursor field will always be greater than or equal to itself, causing it to pull the latest row multiple times until there is new data at the source ([source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append#inclusive-cursors)). 3. Cursor Granularity: If the cursor field's granularity isn't fine enough, you might see the same data being pulled multiple times. For example, if the cursor field only has the granularity of a day, and you run the sync multiple times in a day, the source won't know which records have already been replicated ([source](https://docs.airbyte.com/using-airbyte/core-concepts/sync-modes/incremental-append#inclusive-cursors)). 4. API Limitations: Some APIs might not support filtering by the cursor field, or might always return a certain number of recent records regardless of the cursor field. Check the API documentation to see if there are any such limitations. If you've checked all these points and are still seeing issues, it might be a good idea to reach out to the Airbyte community or support for further assistance.