@kapa.ai Context We have a custom declarative connector built in Connector Builder (Airbyte v.2) that fetches data from the RTE Open API (French electricity TSO). The connector uses a DatetimeBasedCursor with: - step: P1D - lookback_window: P3D - cursor_field: start_date (top-level field on each API record, represents the start of a French energy day) - start_time_option injects start_date as a request parameter - end_time_option injects end_date as a request parameter - start_datetime: {{ config["start_date"] }} - end_datetime: {{ now_utc().strftime('%Y-%m-%dT%H:%M:%SZ') }} The RTE API returns data for a given [start_date, end_date] window. Each energy day runs from 00:00 UTC to 00:00 UTC next day (96 × 15-minute slots). Each slot has its own start_date and end_date inside a values JSON array on the record. The problem The last 15-minute slot of every energy day (starting at 23:45 UTC, ending at 00:00 UTC the next day) is never returned by the API. It appears to be excluded because its end time equals the query's end_date boundary exactly — the API seems to apply a strict < condition on slot end time rather than <=. This slot cannot be recovered via the lookback window either, because: - The current day's window is [00:00 UTC day N, 00:00 UTC day N+1) — the slot's end (00:00 UTC day N+1) hits the boundary and is excluded - The next day's window starts at 00:00 UTC day N+1 — the slot's start (23:45 UTC day N) is before the window, so it's also excluded The slot falls permanently in a gap between two consecutive step windows. What we want We'd like to extend the end_date request parameter by 15 minutes beyond the step's end_time, so the boundary slot is captured within the current step's window. Something like: end_time_option: type: RequestOption field_name: end_date inject_into: request_parameter value: "{{ (stream_interval['end_time'] + duration('PT15M')).strftime('%Y-%m-%dT%H:%M:%SZ') }}" Questions 1. Is the stream_interval['end_time'] + duration('PT15M') syntax correct in the declarative manifest? 2. Is there a recommended pattern for adding a fixed offset to the injected end_date parameter? 3. Could using step: P1DT15M instead be a cleaner alternative, and if so, how would the cursor advance to avoid gaps in the step windows?

I’m afraid the knowledge sources don’t explicitly document adding an offset to

stream_interval.end_time

or show examples of using

duration()

in that context, so I can’t fully confirm the exact syntax you proposed. I’ll outline what is documented and where the gaps are. ### 1. Is

stream_interval['end_time'] + duration('PT15M')

valid? The docs and forum posts confirm: - You can use

stream_interval.start_time

and

stream_interval.end_time

in Jinja expressions for custom parameter injection. [[Custom filter example](https://discuss.airbyte.io/t/5453)] - You can use helper functions like

today_utc()

and

now_utc()

and format them with

strftime

. [[Passing current datetime](https://discuss.airbyte.io/t/5320); [DatetimeBasedCursor ref](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] However, none of the provided docs or forum posts show: - A

duration()

Jinja helper, or - Direct arithmetic on

stream_interval.end_time

(e.g.,

+ duration('PT15M')

). Because of that, I cannot confirm that:

jinja {{ (stream_interval['end_time'] + duration('PT15M')).strftime('%Y-%m-%dT%H:%M:%SZ') }}

is valid in a declarative manifest; the available sources simply don’t document this pattern. ### 2. Recommended pattern for adding a fixed offset? The docs show two main patterns for end times: - Use a fixed or config-based datetime:

yaml   end_datetime: "2021-03-01T00:00:00.000000+0000"

- Use

now_utc()

or

day_delta()

in `end_datetime`:

yaml   end_datetime: "{{ now_utc() }}"   {{ day_delta(-1) }}

[[DatetimeBasedCursor ref](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] And for custom parameter injection, they show using

stream_interval.start_time

/

end_time

directly in query/body parameters, but without offsets. [[Custom filter example](https://discuss.airbyte.io/t/5453)] There is no explicit example of “add a fixed offset to the injected end time” in the docs, so I can’t point to an officially recommended pattern for this. ### 3. Using

step: P1DT15M

instead The documented behavior of

DatetimeBasedCursor

is: - The time range

[start_datetime, end_datetime]

is partitioned into windows according to

step

, with

cursor_granularity

defining the smallest increment and ensuring no overlap. [[Incremental syncs – DatetimeBasedCursor](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/incremental-syncs); [cursor_granularity description](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] - Example with

step: P1D

and

cursor_granularity: PT1S

produces daily windows like: -

2022-01-01T00:00:00

→

2022-01-01T23:59:59

-

2022-01-02T00:00:00

→

2022-01-02T23:59:59

etc. [[Incremental syncs – DatetimeBasedCursor](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/incremental-syncs)] The docs do not show an example with a non‑integral day step like

P1DT15M

, nor do they explain how that would interact with your API’s semantics (which are

[start_date, end_date)

or effectively

< end_date

). So: - From the docs, we know

step

is an ISO 8601 duration and can be things like

PT1H

,

P1D

,

P1M

, etc. [[DatetimeBasedCursor ref](https://docs.airbyte.com/platform/2.0/connector-development/config-based/understanding-the-yaml-file/reference#/definitions/DatetimeBasedCursor)] - But there is no explicit guidance on using

P1DT15M

or how that would avoid your specific “boundary slot” gap with an API that uses a strict

< end_time

filter. Because of that, I cannot confidently recommend

step: P1DT15M

as a “cleaner alternative” based solely on the provided documentation. --- ### Summary Based on the available sources: -

stream_interval.start_time

/

end_time

are valid variables and can be used in custom parameter injection, but adding a duration offset like

duration('PT15M')

is not documented, so the exact syntax you proposed cannot be confirmed. [[Custom filter example](https://discuss.airbyte.io/t/5453)] - There is no documented, recommended pattern for “end_time + fixed offset” in the declarative YAML. -

step

and

cursor_granularity

are documented and must be ISO 8601 durations, but using

P1DT15M

and how that would affect your boundary behavior is not covered in the docs. Given these gaps, the safest next step would be to: - Experiment in a non‑production environment with: - A custom query parameter using

stream_interval.end_time

alone (to confirm baseline behavior), and - If supported in your environment, test whether

duration('PT15M')

is recognized at all. - If that fails, consider adjusting the API window semantics (e.g., shifting

start_datetime

/

end_datetime

slightly) rather than relying on undocumented Jinja helpers. If you can share the exact Builder UI configuration snippet (without secrets), I can help you align it as closely as possible with the documented patterns.

@kapa.ai Context: I have a DatetimeBasedCursor with daily partitions for an API where the day boundary is exactly at UTC midnight. The API uses inclusive filtering: it returns slots where start_date >= request.start_time AND end_date <= request.end_time. Each day has 96 × 15-minute slots, with the first slot starting at 000000Z and the last ending at 000000Z (next day). The problem: I need the HTTP request parameters to be exactly: - start_time = 2025-01-02T000000Z (midnight, inclusive) - end_time = 2025-01-03T000000Z (midnight of next day, inclusive) With step: P1D + cursor_granularity: PT1S: - start_time_option injects stream_slice.start_time = 000000Z ✓ - end_time_option injects stream_slice.end_time - PT1S = 235959Z ✗ (misses last slot) With step: P1DT1S + cursor_granularity: PT1S: - end_time_option injects stream_slice.end_time - PT1S = 000000Z ✓ - start_time_option injects stream_slice.start_time = 000001Z for partition 2+ ✗ (misses first slot) Removing cursor_granularity entirely throws: "If step is defined, cursor_granularity should be as well". Question: With DatetimeBasedCursor, is there a supported way to inject stream_slice.start_time and stream_slice.end_time as raw partition boundaries (without cursor_granularity being subtracted from end_time) into the HTTP request? For example, does using {{ stream_slice.end_time }} in a manual request_parameter (instead of end_time_option) bypass the cursor_granularity subtraction?

A message from kapa.ai