Hari Haran R
09/26/2024, 5:48 AMuser
09/26/2024, 5:48 AMstream_interval
object to dynamically set the start and end time for your data sync. The stream_interval
object represents the current stream interval being processed. The default keys are start_time
and end_time
. Here is an example of how it looks: json { "start_time": "2020-01-01 00:00:00.000+00:00", "end_time": "2020-01-02 00:00:00.000+00:00" }
(Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/variables/stream_interval)) If you need to inject these values into a nested key in a JSON body, you can use the generic parameter section at the bottom of the stream configuration form. This allows you to freely configure query parameters, headers, and properties of the JSON body using jinja expressions and available variables. For example, if you need to set both start and end time in a query
parameter, you can use the stream_interval
variable to configure a query parameter with "key" query
and "value" last_event_time BETWEEN TIMESTAMP "{{stream_interval.start_time}}" AND TIMESTAMP "{{stream_interval.end_time}}"
to filter down to the right window in time. (Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#custom-parameter-injection)) However, if you're trying to inject start/end time into a nested key in JSON body, currently there's a discussion on the Airbyte forum about this issue, but no solution has been provided yet. (Source: [Airbyte Forum](https://discuss.airbyte.io/t/4503))Hari Haran R
09/26/2024, 5:52 AMuser
09/26/2024, 5:52 AMuser
09/26/2024, 5:52 AMday_delta
macro in Airbyte to get the date for one day before the current date. Here is how you can do it: python '{{ day_delta(-1) }}'
This will return the datetime of now()
minus 1 day. (Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#macros))Hari Haran R
09/26/2024, 5:53 AMuser
09/26/2024, 5:53 AMuser
09/26/2024, 5:54 AMday_delta
macro. However, the day_delta
macro returns a datetime string, not a Unix timestamp. Here's how the day_delta
macro works: python '{{ day_delta(-1) }}' -> '2021-08-31T00:00:00.000000+0000'
(Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#macros)) So, if your API expects Unix timestamps, you might need to convert the datetime string to a Unix timestamp. Unfortunately, the provided knowledge sources do not include information on how to do this conversion within Airbyte. Also, the end
time is hardcoded as a Unix timestamp in your JSON. If you want to make it dynamic (e.g., to always be the current time), you would need to use a suitable macro or method, but the provided knowledge sources do not include information on how to do this within Airbyte.Hari Haran R
09/26/2024, 5:59 AMuser
09/26/2024, 5:59 AMuser
09/26/2024, 5:59 AMtoday_utc()
macro is not mentioned in the provided knowledge sources, so I can't confirm if it's available in Airbyte. However, you can use the day_delta
macro to get the current date: python '{{ day_delta(0) }}'
This will return the datetime of now()
. (Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#macros)) So, your JSON would look like this: json { "daterange": "custom", "daterangecustom": { "start": "1726801085", "end": "{{ day_delta(0) }}" } }
Please note that the day_delta
macro returns a datetime string, not a Unix timestamp. If your API expects Unix timestamps, you might need to convert the datetime string to a Unix timestamp. Unfortunately, the provided knowledge sources do not include information on how to do this conversion within Airbyte.Hari Haran R
09/26/2024, 6:17 AMuser
09/26/2024, 6:17 AMuser
09/26/2024, 6:17 AMHari Haran R
09/26/2024, 6:24 AMuser
09/26/2024, 6:24 AMuser
09/26/2024, 6:25 AMstream_interval
object to dynamically set the start time for your data sync. The stream_interval
object represents the current stream interval being processed. The default keys are start_time
and end_time
. Here is an example of how it looks: json {{ "start_time": "2020-01-01 00:00:00.000+00:00", "end_time": "2020-01-02 00:00:00.000+00:00" }}
(Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#/variables/stream_interval)) If you need to inject these values into a nested key in a JSON body, you can use the generic parameter section at the bottom of the stream configuration form. This allows you to freely configure query parameters, headers, and properties of the JSON body using jinja expressions and available variables. For example, if you need to set both start and end time in a query
parameter, you can use the stream_interval
variable to configure a query parameter with "key" query
and "value" last_event_time BETWEEN TIMESTAMP "{{stream_interval.start_time}}" AND TIMESTAMP "{{stream_interval.end_time}}"
to filter down to the right window in time. (Source: [Airbyte Documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#custom-parameter-injection)) However, the stream_interval
object provides the start time in a datetime format, not as a Unix timestamp. If your API expects Unix timestamps, you might need to convert the datetime string to a Unix timestamp. Unfortunately, the provided knowledge sources do not include information on how to do this conversion within Airbyte.