I have an endpoint which provides a field updatedA...
# replication-ask-ai
l
I have an endpoint which provides a field updatedAt in the form of epoch seconds (integer). How can I use this field to build incremental sync using datetime based cursor?
k
You can use
DatetimeBasedCursor
with Unix epoch seconds by configuring the datetime format as
%s
(Unix timestamp in seconds). Airbyte explicitly supports this both in the YAML/low-code CDK and in the Connector Builder UI.[\[DatetimeBasedCursor ref\]](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)[\[Builder incremental\]](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/incremental-sync) ### If you’re using a declarative (manifest/YAML) connector In your
DeclarativeStream
, configure
incremental_sync
like this (adapt field names as needed):
Copy code
yaml incremental_sync:   type: DatetimeBasedCursor   cursor_field: "updatedAt"          # your epoch-seconds field   datetime_format: "%s"              # cursor is Unix timestamp (seconds)   cursor_granularity: "PT1S"         # 1-second granularity   start_datetime:     datetime: "{{ config['start_date'] }}"     datetime_format: "%Y-%m-%d %H:%M:%S"   end_datetime:     datetime: "{{ today_utc() }}"     datetime_format: "%Y-%m-%d"   step: "P1D"                        # window size (e.g. 1 day)   start_time_option:     field_name: "updated_after"      # how your API expects the start time     inject_into: "request_parameter"   end_time_option:     field_name: "updated_before"     # how your API expects the end time     inject_into: "request_parameter"
This pattern is the same as the example used for Unix timestamps in the Delighted connector.[\[Forum UNIX timestamps\]](https://discuss.airbyte.io/t/3921) ### If you’re using the Connector Builder UI In the Incremental sync section for your stream: 1. Cursor field: set to
updatedAt
. 2. Cursor datetime formats (or “Datetime format” in older UI): add
%s
so Airbyte parses
updatedAt
as Unix seconds.[\[Builder incremental\]](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/incremental-sync) 3. Configure Start datetime and End datetime (usually “now”) in a human-readable format (e.g.
%Y-%m-%dT%H:%M:%S
). 4. Under Inject start/end time into outgoing HTTP request, map those times to your API’s query/body parameters (e.g.
updated_after
,
updated_before
). With
%s
as the cursor format, Airbyte will correctly track and advance the incremental state using your integer epoch
updatedAt
field.
l
@kapa.ai I was able to follow until Step 3 in the connector builder UI. But the API doesnt have a query/body parameters to inject start/end time into outgoing HTTP requests. Will it still work?
k
A message from kapa.ai
It depends on how your API returns data. From the docs, incremental sync in the Connector Builder assumes either: 1. The API lets you filter by time (via query/body params, headers, or path), or 2. The API behaves like a time-ordered feed (newest → oldest) where you can’t filter by time, but you can page through results. [[Incremental sync](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/incremental-sync)][[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] If your API does not have any way to pass start/end time: 1. In the Incremental sync section, set API time filtering capabilities to “No filter”. This disables the start/end time injection options. [[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] 2. Configure pagination so that Airbyte can fetch all pages. 3. Keep
updatedAt
as the cursor field (with
%s
as the datetime format). 4. Ensure the endpoint returns records sorted from newest to oldest across pages. Airbyte will: - Read pages in order, - Stop when it reaches records whose
updatedAt
is ≤ the stored cursor, - Save the latest
updatedAt
as state for the next sync. [[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] If the data is not sorted newest → oldest, the docs recommend not using incremental sync and instead doing full refreshes on a schedule. [[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)]
l
@kapa.ai Are you asking me to use incrementing count cursor instead of datetime based cursor?
k
A message from kapa.ai
No, you can still use a datetime-based cursor with your
updatedAt
epoch field; you do not need to switch to an incrementing count cursor. From the docs, there are two incremental patterns in the Connector Builder: [[Builder incremental](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/incremental-sync)][[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] 1. Range mode (with time filtering) - You inject start/end datetimes into the request (query/body/header/path). - This is the part you can’t use because your API doesn’t accept time filters. 2. “No filter” mode (incremental sync without time filtering) - For APIs that do not allow filtering by date, but return a paginated feed sorted from newest to oldest. - You still configure a cursor field (your
updatedAt
in epoch seconds, using
%s
as the datetime format). - Airbyte keeps reading pages until it encounters records whose cursor value is ≤ the stored cursor, then stops and saves the latest cursor for the next sync. [[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] So in your case: - Keep using Datetime-based cursor on
updatedAt
(epoch seconds with
%s
). - Set API time filtering capabilities to “No filter”. - Make sure the endpoint returns data **newest → oldest across pages**; otherwise, the docs recommend disabling incremental and using full refresh instead. [[No filter mode](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)]
l
@kapa.ai I dont see any option to Set API time filtering capabilities to “No filter”.
k
A message from kapa.ai
You’re right to be confused here — the docs clearly describe a “No filter” / “API time filtering capabilities” option for incremental sync in the Connector Builder UI, but they don’t show exactly where it appears in the current UI, and the knowledge sources don’t include updated screenshots or step‑by‑step UI navigation. [[Incremental sync 1.7](https://docs.airbyte.com/platform/1.7/connector-development/connector-builder-ui/incremental-sync)][[No filter mode 1.8](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] From the docs we know: - There is a configuration called “API time filtering capabilities” with a “No filter” option, intended exactly for APIs that don’t accept start/end time but return a paginated feed sorted newest → oldest. [[No filter mode 1.8](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] - When “No filter” is selected, the Builder: - Disables “Inject start/end time into outgoing HTTP request” and “Split up interval”. - Uses the cursor field (your
updatedAt
with
%s
) and pagination, and stops when it reaches records whose cursor ≤ the stored cutoff. [[No filter mode 1.8](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] However, the knowledge sources do not show: - Whether this option has been renamed, moved, or hidden in the latest Builder UI. - Where exactly in the current UI you should click to change it. So, based only on the docs: 1. Conceptually, you should be able to configure your stream as: - Cursor field:
updatedAt
- Cursor datetime format:
%s
(epoch seconds) [[Builder incremental (next)](https://docs.airbyte.com/platform/next/connector-development/connector-builder-ui/incremental-sync)] - API time filtering capabilities: “No filter” (so no start/end injection is required) [[No filter mode 1.8](https://docs.airbyte.com/platform/1.8/connector-development/connector-builder-ui/incremental-sync#incremental-sync-without-time-filtering)] 2. If you don’t see that option in your UI, the knowledge sources don’t explain why (version difference, feature flag, or UI change), so I can’t reliably tell you how to expose it from within the product. Given these limits, your next concrete steps (based on the docs) are: - Double‑check you’re in the Incremental sync configuration for the stream (not just pagination or schema). That’s where “API time filtering capabilities” is documented to live. [[Incremental sync 1.7](https://docs.airbyte.com/platform/1.7/connector-development/connector-builder-ui/incremental-sync)] - If the “API time filtering capabilities” / “No filter” control truly isn’t present, this is outside what the current docs cover. In that case, you’ll likely need to: - Either use a manifest/low‑code YAML connector where you can explicitly set
is_data_feed
/
is_client_side_incremental
on a
DatetimeBasedCursor
, which is the config‑file equivalent of “No filter”; [[DatetimeBasedCursor ref](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] - Or contact Airbyte support / community with your Builder version and screenshots, since the docs don’t describe the UI state you’re seeing. So: the behavior you want is supported conceptually (datetime cursor with no time filters, using a feed), but the knowledge sources don’t contain enough information to explain why you can’t see the “API time filtering capabilities / No filter” option in your specific Builder UI.
l
@kapa.ai If I provide you the necessary details, can you provide the low-code yaml that I can use directly? URL - https://api.holded.com/api/invoicing/v1/contacts API_KEY - XYZ Use updatedAt for incremental sync cursor
k
A message from kapa.ai
I’m sorry, but the knowledge sources don’t contain connector-specific examples for the Holded API, so I can’t confidently generate a fully correct, ready‑to‑run manifest for that exact endpoint (URL, auth header name, response shape, pagination style, etc. are not documented in the provided sources). I can only show the pattern for using
updatedAt
(epoch seconds) as a datetime cursor in a low‑code YAML connector. Below is a template snippet that demonstrates how to wire a
DatetimeBasedCursor
to an
updatedAt
field expressed as Unix seconds, using an API key in a header and simple “single page” pagination. You will need to adapt the parts marked with comments (
# TODO
) to match the real Holded API (auth header name, pagination params, response field names, etc.), because those details are not in the knowledge sources. This pattern is based on: -
DatetimeBasedCursor
schema and properties (including
cursor_field
,
cursor_datetime_formats
,
start_datetime
,
step
,
lookback_window
, etc.). [[Datetime cursor ref](https://docs.airbyte.com/platform/1.8/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] - How incremental syncs with datetime cursors work in low‑code connectors. [[Incremental syncs](https://docs.airbyte.com/platform/1.8/connector-development/config-based/understanding-the-yaml-file/incremental-syncs#datetimebasedcursor)] - Handling Unix timestamps via
%s
in
cursor_datetime_formats
, as shown in the Delighted example. [[Forum UNIX timestamps](https://discuss.airbyte.io/t/3921)]
Copy code
yaml version: "0.78.0"  # TODO: set to the Airbyte spec version you are targeting  definitions:   base_requester:     type: HttpRequester     url_base: "<https://api.holded.com/api/invoicing/v1/>"     authenticator:       type: ApiKeyAuthenticator       api_token: "{{ config['api_key'] }}"       # TODO: confirm the correct header name for Holded; this is a guess       inject_into:         type: RequestOption         field_name: "Authorization"         inject_into: "header"    contacts_stream:     type: DeclarativeStream     name: "contacts"     primary_key: "id"  # TODO: adjust to actual primary key field     retriever:       type: SimpleRetriever       requester:         $ref: "#/definitions/base_requester"         path: "contacts"         http_method: "GET"         request_parameters: {}         request_headers: {}       paginator:         type: NoPagination  # TODO: replace with proper paginator if the API is paginated     schema_loader:       type: InlineSchemaLoader       schema:         type: object         additionalProperties: true         properties:           id:             type: string           updatedAt:             type: integer           # TODO: add other fields as needed      incremental_sync:       type: DatetimeBasedCursor       cursor_field: "updatedAt"       # updatedAt is epoch seconds       cursor_datetime_formats:         - "%s"       cursor_granularity: "PT1S"       # earliest date to sync from, provided by user       start_datetime:         datetime: "{{ config['start_date'] }}"         datetime_format: "%Y-%m-%dT%H:%M:%S"       # end time is "now" in UTC       end_datetime:         datetime: "{{ today_utc() }}"         datetime_format: "%Y-%m-%dT%H:%M:%S"       step: "P1D"  # 1‑day windows       # No start_time_option / end_time_option here because the docs you provided       # don’t show how Holded accepts time filters; this assumes a data‑feed style       # where the API is read fully and the cursor is applied client‑side.       is_client_side_incremental: true  streams:   - $ref: "#/definitions/contacts_stream"  check:   type: CheckStream   stream_names:     - "contacts"  spec:   connectionSpecification:     type: object     required:       - api_key       - start_date     properties:       api_key:         type: string         title: API Key         description: "Holded API key"         airbyte_secret: true       start_date:         type: string         title: Start date         description: "Earliest updatedAt to sync from (e.g. 2023-01-01T00:00:00)"
Key points you’ll likely need to verify/adjust based on the real API (not covered in the knowledge sources): - Auth header: whether Holded expects
Authorization: XYZ
,
X-API-KEY: XYZ
, or something else. - Pagination: if
/contacts
is paginated, replace
NoPagination
with the appropriate paginator (e.g. cursor or page/limit). - Time filtering: if the API does support
updatedAt
filters (e.g.
updatedAtFrom
/
updatedAtTo
), you can add
start_time_option
and
end_time_option
to the
DatetimeBasedCursor
as shown in the Unix‑timestamp example from the forum. [[Forum UNIX timestamps](https://discuss.airbyte.io/t/3921)] - Schema: adjust
primary_key
and properties to match the actual response. Because the Holded‑specific details are not in the provided docs, please treat this as a starting template rather than a drop‑in manifest.
l
@kapa.ai Can you also share how to implement incrementing count cursor for incremental sync?
k
A message from kapa.ai
You can implement an incrementing count cursor in a low‑code (config‑based) connector using the
IncrementingCountCursor
component. This cursor assumes a monotonically increasing integer field (e.g.
id
,
offset
, etc.) and uses it as the bookmark for incremental syncs. [[Incrementing cursor](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/IncrementingCountCursor)] ### 1. When to use
IncrementingCountCursor
Use it when: - Your records have a top‑level integer field that always increases (e.g.
id
,
sequence
,
offset
). - The API returns records **sorted ascending by that field**; the docs explicitly require this to avoid data loss. [[Incrementing cursor](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/IncrementingCountCursor)] ### 2. Core configuration fields From the YAML reference: [[Incrementing cursor](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/IncrementingCountCursor)] -
cursor_field
(string, required) Top‑level field used as the bookmark (e.g.
id
). -
allow_catalog_defined_cursor_field
(boolean, optional) If
true
, lets users override
cursor_field
from the connection catalog. -
start_value
(string or integer, required) Earliest value to sync from (e.g.
0
or
{{ config['start_value'] }}
). -
start_value_option
(RequestOption, optional) How to send the
start_value
to the API (e.g. as a query parameter). ### 3. Example: using
id
as an incrementing cursor Below is a minimal example of wiring
IncrementingCountCursor
into a declarative stream:
Copy code
yaml definitions:   my_stream:     type: DeclarativeStream     name: my_stream     primary_key: id     retriever:       type: SimpleRetriever       requester:         type: HttpRequester         url_base: "<https://api.example.com/>"         path: "items"         http_method: "GET"       paginator:         type: NoPagination     schema_loader:       type: InlineSchemaLoader       schema:         type: object         additionalProperties: true         properties:           id:             type: integer      incremental_sync:       type: IncrementingCountCursor       cursor_field: "id"       allow_catalog_defined_cursor_field: false       start_value: 0       # Optional: if the API accepts a "min_id" parameter       start_value_option:         type: RequestOption         field_name: "min_id"         inject_into: "request_parameter"
Key points from the docs applied here: [[Incrementing cursor](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/IncrementingCountCursor)] -
cursor_field
is top‑level and integer. -
start_value
defines the first record to pull. -
start_value_option
(optional) shows how you’d inject that into the request if the API supports it. If your API doesn’t accept a filter parameter, you can omit `start_value_option`; Airbyte will still track state client‑side, but you must ensure the API always returns records in ascending order by
cursor_field
to avoid missing or duplicating records.
l
@kapa.ai How to adda atransformed date field to the records where the epoch is in readable format
k
A message from kapa.ai
You can add a new field with the human‑readable date by using an AddFields / Adding fields transformation and a Jinja expression that converts the epoch value. There are two main ways, depending on how you’re building the connector. --- ## 1. In the Connector Builder UI Use the Transformations → Adding fields feature: [[Adding fields](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/record-processing#adding-fields)] 1. Go to your stream → Record processing → Transformations. 2. Add a new transformation of type add. 3. Path: where you want the readable date, e.g.
["readable_updated_at"]
. 4. Value: a Jinja expression that transforms your epoch field. If your epoch is already a string or integer field on the record (e.g.
record['updatedAt']
), you can follow the same pattern as the hashing example, but using a date conversion filter if available in your environment. The docs show the general pattern for using filters like:
Copy code
jinja {{ record['from']['name'] | hash('md5') }}
for hashing; you’d replace
hash('md5')
with the appropriate date conversion filter supported by your runtime. [[Adding fields](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/record-processing#adding-fields)] However, the provided knowledge sources do not document a built‑in Jinja filter that converts epoch seconds to a formatted date (only
hash
is explicitly shown), so I cannot give you a guaranteed‑working exact expression for epoch→date in the Builder UI. --- ## 2. In a low‑code (YAML) declarative manifest Use the
AddFields
transformation with an
AddedFieldDefinition
. [[Record selector transformations](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/record-selector#transformations)] Example structure:
Copy code
yaml transformations:   - type: AddFields     fields:       - path: ["readable_updated_at"]         value: "{{ record['updatedAt'] }}"
-
path
is where the new field will be added (top‑level or nested). [[Record selector transformations](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/record-selector#transformations)] -
value
is a string expression; you can reference the original epoch field via
record['updatedAt']
. [[AddedFieldDefinition](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#addedfielddefinition)] Again, the docs show how to copy or hash existing fields, but they do not document a built‑in function to convert epoch seconds to a formatted date directly in YAML (e.g. no example of
| to_datetime
or similar), so I can’t provide a confirmed expression that will both parse epoch and format it. --- ### Summary - You can add a new field derived from your epoch field using Add / AddFields transformations in both the Builder UI and YAML. [[Adding fields](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/record-processing#adding-fields); [Record selector transformations](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/record-selector#transformations)] - The knowledge sources show how to: - Reference existing fields (
record[...]
) and - Apply filters like
hash('md5')
. [[Adding fields](https://docs.airbyte.com/platform/connector-development/connector-builder-ui/record-processing#adding-fields); [AddedFieldDefinition](https://docs.airbyte.com/platform/connector-development/config-based/understanding-the-yaml-file/reference#addedfielddefinition)] - They do not explicitly document a built‑in Jinja filter for converting epoch seconds to a human‑readable date, so I cannot give a fully certain epoch→date expression; you may need to: - Do the epoch→date conversion downstream (e.g. in your warehouse), or - Implement a custom Python CDK connector and use a custom transform in code. [[Custom schema transform](https://docs.airbyte.com/platform/connector-development/cdk-python/schemas#custom-schema-type-transformation)]