@kapa.ai i want to inject a value into an outgoing http request in the connection builder. My request is a post request with a json body like this:

{
  "Source": "Integrator",
  "Data": {
    "ApiDocumentId": "3f134d8f-1f72-4c77-b017-0006a52e6ba2",
    "Request": {
      "Filters": [
        {
          "Field": "RecModified",
          "Operator": "Less than or equal",
          "Value": "2024-09-18T12:55:52.937"
        }
      ],
      "Top": 5
      "Limit": 10
    }
  }
}

I want to inject a value for

Limit

as part of pagination. What key name would i use to nest the field inside the json object?

To inject a value for the

Limit

field nested within the JSON body of your POST request in the Airbyte Connector Builder, you'll need to use a specific format for the key name to represent the nested structure. Based on the information provided in the knowledge sources, here's how you can approach this: 1. In the Connector Builder UI, you'll need to use the custom parameter injection method, as the standard pagination form doesn't support nested fields directly. 2. To represent the nested path to the

Limit

field, you should use dot notation. The key name you would use is:

Data.Request.Limit

This key represents the path to the

Limit

field within your nested JSON structure. To implement this in the Connector Builder: 1. Disable the standard injection options in the pagination form. 2. Scroll to the generic parameter section at the bottom of the stream configuration form. 3. Configure a parameter for the JSON body using a Jinja expression to inject the value. Here's an example of how you might set this up: - Inject into:

body_json

- Field name:

Data.Request.Limit

- Value:

{{ config.page_size }}

This setup will inject the configured page size into the

Limit

field of your nested JSON structure. Remember that you may need to adjust other parts of your JSON body to be dynamic as well, especially for pagination purposes. For example, you might want to update the

RecModified

filter value in subsequent requests. For more details on custom parameter injection and pagination in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection).

@kapa.ai when i do

Data.Request.Limit

it just injects it at the root level of the body like this:

{
    "Source": "Integrator",
    "Data": {
      "ApiDocumentId": "3f134d8f-1f72-4c77-b017-0006a52e6ba2",
      "Request": {
        "Filters": [
          {
            "Field": "RecModified",
            "Operator": "Less than or equal",
            "Value": "2024-09-18T12:55:52.937"
          }
        ],
        "Top": 5
      }
    },
    "Data.Request.Limit": 20,
  },

A message from kapa.ai

I apologize for the confusion, but currently, Airbyte's Connector Builder UI does not support injecting values into nested fields within a JSON body directly. The dot notation (

Data.Request.Limit

) will not work as expected and will instead create a new field at the root level of the JSON body, as you've observed. As a workaround, you can use a custom method to handle this. You can disable the standard injection options in the pagination form and use the generic parameter section at the bottom of the stream configuration form to freely configure properties of the JSON body using Jinja expressions and available variables. However, this method might not work for deeply nested fields. Here's an example of how you might set this up: - Inject into:

body_json

- Field name:

Data

- Value:

{"ApiDocumentId": "3f134d8f-1f72-4c77-b017-0006a52e6ba2", "Request": {"Filters": [{"Field": "RecModified", "Operator": "Less than or equal", "Value": "2024-09-18T12:55:52.937"}], "Top": 5, "Limit": {{ config.page_size }}}

This setup will inject the entire

Data

object, including the dynamic

Limit

value, into the JSON body. However, you may need to adjust this to suit your needs, especially if other parts of your JSON body need to be dynamic. For more details on custom parameter injection and pagination in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection).

I need to do the same with a date field for incremental syncs. How can i use the custom parameter ingestion to inject the start and end datetime (for range) into the body like this:

{
  "Source": "Integrator",
  "Data": {
    "Request": {
      "Filters": [
        {
          "Field": "RecModified",
          "Operator": "Less than or equal",
          "Value": "X?"
        },
        {
          "Field": "RecModified",
          "Operator": "More than or equal",
          "Value": "Y?"
        }
      ],
    }
  }
}

A message from kapa.ai

To inject the start and end datetime into the JSON body for incremental syncs, you can use the

stream_interval.start_time

and

stream_interval.end_time

variables provided by Airbyte. These variables represent the start and end of the time range for the current sync. Here's how you can set this up in the Connector Builder UI: - Inject into:

body_json

- Field name:

Data

- Value:

json {   "Request": {     "Filters": [       {         "Field": "RecModified",         "Operator": "Less than or equal",         "Value": "{{ stream_interval.end_time }}"       },       {         "Field": "RecModified",         "Operator": "More than or equal",         "Value": "{{ stream_interval.start_time }}"       }     ]   } }

This setup will inject the entire

Data

object, including the dynamic start and end datetime values, into the JSON body. The

stream_interval.start_time

and

stream_interval.end_time

variables will be replaced with the actual start and end times for the current sync. Please note that the datetime format of the

stream_interval.start_time

and

stream_interval.end_time

variables should match the format expected by your API. If necessary, you can use Jinja's

strftime

function to format these datetime values. For more details on custom parameter injection and incremental syncs in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#custom-parameter-injection) and the [Airbyte documentation on incremental syncs](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration).

@kapa.ai What jinja can i use to set the limit from offset increment pagination

A message from kapa.ai

To set the limit for offset increment pagination using Jinja in Airbyte, you can use the

config.page_size

variable. This variable represents the page size that you've configured in the Connector Builder UI. Here's an example of how you can use this variable: - Inject into:

request_parameter

(or

body_json

if the limit is part of the request body) - Field name:

limit

(or the appropriate field name for your API) - Value:

{{ config.page_size }}

This setup will inject the configured page size into the

limit

field of your outgoing HTTP request. For more details on using Jinja expressions in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection).

@kapa.ai when i do

"Filters": [
      {
        "Field": "RecModified",
        "Operator": "Less than or equal",
        "Value": "{{ stream_interval.end_time }}"
      }
    ],

I get

ERROR
Something went wrong in the connector. See the logs for more details. - Traceback (most recent call last):
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 133, in read
    yield from self._read_stream(
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 239, in _read_stream
    for record_data_or_message in record_iterator:
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/streams/core.py", line 192, in read
    for record_data_or_message in records:
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 137, in read_records
    yield from self.retriever.read_records(self.get_json_schema(), stream_slice)  # type: ignore # records are of the correct type
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 378, in read_records
    for stream_data in self._read_pages(record_generator, self.state, _slice):
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 301, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 487, in _fetch_next_page
    return self.requester.send_request(
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 305, in send_request
    request, response = self._http_client.send_request(
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 406, in send_request
    response: requests.Response = self._send_with_retry(
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 236, in _send_with_retry
    response = backoff_handler(rate_limit_backoff_handler(user_backoff_handler))(request, request_kwargs, log_formatter=log_formatter, exit_on_rate_limit=exit_on_rate_limit)  # type: ignore # mypy can't infer that backoff_handler wraps _send
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 290, in _send
    self._handle_error_resolution(
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 370, in _handle_error_resolution
    raise DefaultBackoffException(
airbyte_cdk.sources.streams.http.exceptions.DefaultBackoffException: Internal server error.
ERROR
None - Traceback (most recent call last):
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/connector_builder/message_grouper.py", line 302, in _read_stream
    yield from AirbyteEntrypoint(source).read(source.spec(self.logger), config, configured_catalog, state)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/entrypoint.py", line 191, in read
    for message in self.source.read(self.logger, config, catalog, state):
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/manifest_declarative_source.py", line 165, in read
    yield from super().read(logger, config, catalog, state)
  File "/home/airbyte/.pyenv/versions/3.10.14/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 177, in read
    raise AirbyteTracedException(message=error_message, failure_type=FailureType.config_error)
airbyte_cdk.utils.traced_exception.AirbyteTracedException: None

A message from kapa.ai

The error message you're seeing indicates that there's an issue with the server, specifically an "Internal server error". This typically means that the server encountered an unexpected condition that prevented it from fulfilling the request. The issue might be related to the format of the

stream_interval.start_time

and

stream_interval.end_time

variables. These variables are in ISO 8601 format, which includes date and time information. If your API expects a different format, you might need to convert these variables to the expected format using Jinja's

strftime

function. Here's an example of how you can format these variables:

json {   "Filters": [     {       "Field": "RecModified",       "Operator": "Less than or equal",       "Value": "{{ stream_interval.end_time.strftime('%Y-%m-%d') }}"     },     {       "Field": "RecModified",       "Operator": "More than or equal",       "Value": "{{ stream_interval.start_time.strftime('%Y-%m-%d') }}"     }   ] }

In this example, the

strftime('%Y-%m-%d')

function is used to format the datetime variables to a string in the 'YYYY-MM-DD' format. You should replace

'%Y-%m-%d'

with the datetime format expected by your API. If the issue persists, it might be related to the API you're trying to connect to. You might want to check the API documentation or reach out to the API provider for further assistance. For more details on custom parameter injection and incremental syncs in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#custom-parameter-injection) and the [Airbyte documentation on incremental syncs](https://docs.airbyte.com/connector-development/connector-builder-ui/incremental-sync#configuration).

@kapa.ai are you sure to inject the page size its

{{ config.page_size }}

? When i do that it doesn’t work

A message from kapa.ai

Yes,

{{ config.page_size }}

is the correct Jinja variable to use for injecting the page size in Airbyte's Connector Builder UI. This variable represents the page size that you've configured in the UI. However, please note that the actual field name and injection method can vary depending on the API you're working with. For example, some APIs might expect the page size to be included as a query parameter, while others might expect it to be part of the request body. The field name can also vary; it might be

page_size

,

limit

,

max

, etc. Here's an example of how you can use this variable: - Inject into:

request_parameter

(or

body_json

if the page size is part of the request body) - Field name:

page_size

(or the appropriate field name for your API) - Value:

{{ config.page_size }}

If this setup doesn't work, it might be due to an issue with the API or the specific endpoint you're trying to connect to. You might want to check the API documentation or reach out to the API provider for further assistance. For more details on using Jinja expressions in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection).

@kapa.ai this is what it says on the connection builder

Limit
The number of records to include in each pages.


Variables available for interpolation:
config
response
See macros available for interpolation here
Example values
100
{{ config['page_size'] }}

A message from kapa.ai

Based on the information provided from the connection builder, it seems like the correct Jinja variable to use for injecting the page size is

{{ config['page_size'] }}

. Here's how you can use this variable: - Inject into:

request_parameter

(or

body_json

if the page size is part of the request body) - Field name:

page_size

(or the appropriate field name for your API) - Value:

{{ config['page_size'] }}

This setup will inject the configured page size into the

page_size

field of your outgoing HTTP request. If this setup doesn't work, it might be due to an issue with the API or the specific endpoint you're trying to connect to. You might want to check the API documentation or reach out to the API provider for further assistance. For more details on using Jinja expressions in the Connector Builder UI, you can refer to the [Airbyte documentation on custom parameter injection](https://docs.airbyte.com/connector-development/connector-builder-ui/pagination#custom-parameter-injection).