https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> I am trying to set up an endpoint o...
# replication-ask-ai
s
@kapa.ai I am trying to set up an endpoint on connector builder which has the url path 
/loans:search
and the POST method. When trying to test it, i get the following error:
Copy code
Traceback (most recent call last):
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 152, in read
    yield from self._read_stream(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/abstract_source.py", line 276, in _read_stream
    for record_data_or_message in record_iterator:
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/streams/core.py", line 196, in read
    for record_data_or_message in records:
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/declarative_stream.py", line 154, in read_records
    yield from self.retriever.read_records(self.get_json_schema(), stream_slice)  # type: ignore # records are of the correct type
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 446, in read_records
    for stream_data in self._read_pages(record_generator, self.state, _slice):
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 343, in _read_pages
    response = self._fetch_next_page(stream_state, stream_slice, next_page_token)
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/retrievers/simple_retriever.py", line 574, in _fetch_next_page
    return self.requester.send_request(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/declarative/requesters/http_requester.py", line 342, in send_request
    request, response = self._http_client.send_request(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 524, in send_request
    response: requests.Response = self._send_with_retry(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 270, in _send_with_retry
    response = backoff_handler(rate_limit_backoff_handler(user_backoff_handler))(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/backoff/_sync.py", line 105, in retry
    ret = target(*args, **kwargs)
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 340, in _send
    self._handle_error_resolution(
  File "/home/airbyte/.pyenv/versions/3.10.15/lib/python3.10/site-packages/airbyte_cdk/sources/streams/http/http_client.py", line 427, in _handle_error_resolution
    raise MessageRepresentationAirbyteTracedErrors(
airbyte_cdk.sources.streams.http.http_client.MessageRepresentationAirbyteTracedErrors: Invalid Protocol Schema: The endpoint that data is being requested from is using an invalid or insecure. Exception: requests.exceptions.InvalidSchema
what does this mean? and in which logs can i view the exact text of the request that was run during the test to produce this error?
k
I'm sorry, but the knowledge sources do not contain enough information to give a fully confident answer about this specific error or where to find the exact request logs in the Connector Builder. That said, I can offer some analysis based on what the error message itself tells us: What the Error Means The error at the bottom of the traceback is:
Copy code
airbyte_cdk.sources.streams.http.http_client.MessageRepresentationAirbyteTracedErrors: 
Invalid Protocol Schema: The endpoint that data is being requested from is using an invalid 
or insecure. Exception: requests.exceptions.InvalidSchema
This is a 
requests.exceptions.InvalidSchema
error, which means the URL being constructed for the request is malformed — specifically, the protocol/schema part of the URL is invalid or missing. This is not an API authentication or rate-limit issue. A likely cause with your setup is the URL path 
/loans:search
. The colon (
:
) in the path segment could be causing the URL to be misinterpreted. Depending on how the base URL and path are being concatenated in the Connector Builder, the resulting URL might end up malformed (e.g., 
loans:search
being parsed as a protocol scheme instead of a path). Things to check: • Verify your Base URL is correctly set (e.g., 
https://api.example.com
) and does not have a trailing slash issue when combined with 
/loans:search
. • Try URL-encoding the colon if the API allows it: 
/loans%3Asearch
(though this may change the API behavior). • Double-check that the full constructed URL would be something like 
https://api.example.com/loans:search
and not something broken like 
loans:search
being treated as a scheme. Where to Find Request Logs The knowledge sources do not explicitly document where to view the exact HTTP request text in the Connector Builder test UI. Generally in Airbyte, you would look at: • The connector logs tab in the Connector Builder test panel. • The sync logs in the Airbyte UI if running a full sync. For more detailed request/response logging, you may need to check the Airbyte documentation directly or the Airbyte community forum or GitHub issues for guidance specific to your version. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai this is what my request body JSON payload looks like:
Copy code
{
  "filterCriteria": [
    {
      "field": "lastModifiedDate",
      "operator": "AFTER",
      "value": "{{ config['start_date'] }}"
    }
  ],
  "sortingCriteria": {
    "field": "lastModifiedDate",
    "order": "ASC"
  }
}
In my incremental sync options, i want to inject start time into the Body JSON Payload to replace 
filterCriteria.value
. What do i enter in field path to set this up?
k
Based on the knowledge sources, you can use 
field_path
for nested 
body_json
injection. For your structure, you need to navigate into the 
filterCriteria
array and then to 
value
. However, there is an important limitation here: 
filterCriteria
is an array, and the knowledge sources do not explicitly document how 
field_path
handles array indexing. What is documented is that 
field_path
works for nested JSON objects. For a straightforward nested object case, the 
field_path
in your YAML would look like this:
Copy code
start_time_option:
  type: RequestOption
  inject_into: body_json
  field_path:
    - filterCriteria
    - "0"        # array index - not confirmed to work
    - value
A real-world example from the source-linear connector shows 
field_path
used for nested objects like this: [Linear manifest]
Copy code
start_time_option:
  type: RequestOption
  inject_into: body_json
  field_path:
    - variables
    - filter
    - updatedAt
    - gte
This works cleanly because those are all object keys. [Request Options docs] Recommended Workaround Since your 
filterCriteria
is an array (which may not be supported by 
field_path
), the safer approach is to disable the built-in start/end time injection in the incremental sync form and instead use custom parameter injection directly in your request body JSON, referencing 
stream_interval
variables: [Custom parameter injection]
Copy code
requester:
  type: HttpRequester
  http_method: POST
  request_body_json:
    filterCriteria:
      - field: "lastModifiedDate"
        operator: "AFTER"
        value: "{{ stream_interval.start_time }}"
    sortingCriteria:
      field: "lastModifiedDate"
      order: "ASC"
This approach gives you full control over the body structure, including arrays, and avoids the uncertainty around array indexing in 
field_path
. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai which datetime format is this value in? 
2026-08-10T10:26:11+01:00
k
The value 
2026-08-10T10:26:11+01:00
is in ISO 8601 format with a UTC offset, which corresponds to the format code:
Copy code
%Y-%m-%dT%H:%M:%S%z
Where: • 
%Y
= 
2026
(4-digit year) • 
%m
= 
08
(month) • 
%d
= 
10
(day) • 
%H
= 
10
(hour, 24h) • 
%M
= 
26
(minute) • 
%S
= 
11
(second) • 
%z
= 
+01:00
(UTC offset) This is confirmed by the supported format placeholders in the Airbyte 
DatetimeBasedCursor
documentation, where 
%z
is described as "UTC offset" with examples like 
+0000
and 
-04:00
. [DatetimeBasedCursor reference] So in your incremental sync configuration, you would set:
Copy code
cursor_datetime_formats:
  - "%Y-%m-%dT%H:%M:%S%z"
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai I have another stream I am trying to set up. The parent stream is incremental, but this child has no field that can be used for an incremental cursor. The url path is configured like this: /loans/{{ stream_partition.parentKey }}/schedule What setup can I use so that I only refresh the schedule for those parent records that have been updated in the latest incremental run?
k
This is exactly the use case for combining 
incremental_dependency: true
on the parent stream config with 
global_substream_cursor: true
on the child stream. The knowledge sources actually show this pattern explicitly in the Stripe connector for streams that have no meaningful child-level cursor. [Stripe manifest] The idea is: • 
incremental_dependency: true
— makes the parent re-execution use its persisted cursor state, so only updated parent records are returned as partitions on subsequent syncs. • 
global_substream_cursor: true
— since the child has no real incremental field, a single global cursor is used just to satisfy the incremental sync configuration requirement. The real filtering happens at the parent level. Here's how your YAML would look:
Copy code
loan_schedule:
  type: DeclarativeStream
  retriever:
    type: SimpleRetriever
    requester:
      path: "/loans/{{ stream_partition.parentKey }}/schedule"
      http_method: GET
    partition_router:
      type: SubstreamPartitionRouter
      parent_stream_configs:
        - type: ParentStreamConfig
          stream: "#/definitions/streams/loans"
          parent_key: id
          partition_field: parentKey
          incremental_dependency: true   # only fetch updated parent records
  incremental_sync:
    type: DatetimeBasedCursor
    cursor_field: lastModifiedDate      # use the parent's cursor field if available on child records, or any date field
    global_substream_cursor: true       # single global cursor since child has no meaningful incremental field
The Stripe connector comments explain this pattern well: > "This stream is not truly incremental on the child level. We're configuring it this way to support incremental syncs on the parent only, which helps reduce the size of the state." [Stripe manifest] Key requirement This only works correctly if the parent loan record's cursor field (e.g. 
lastModifiedDate
) is updated whenever its schedule changes. From the docs: > "The API must ensure that the parent record's cursor is updated whenever child records are added or updated. If this requirement is not met, child records added to older parent records will be lost." [Incremental sync docs] If the parent's cursor is not reliably updated when the schedule changes, you would need to fall back to a 
lookback_window
on the parent stream to re-check recently modified records. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai ok my stream config now looks like this:
Copy code
loan_schedule:
      type: DeclarativeStream
      name: loan_schedule
      primary_key:
        - encodedKey
      retriever:
        type: SimpleRetriever
        requester:
          $ref: "#/definitions/base_requester"
          path: /loans/{{ stream_partition.parentKey }}/schedule
          http_method: GET
          request_parameters:
            detailsLevel: FULL
          request_headers:
            Accept: application/vnd.mambu.v2+json
        record_selector:
          type: RecordSelector
          extractor:
            type: DpathExtractor
            field_path:
              - installments
        partition_router:
          type: SubstreamPartitionRouter
          parent_stream_configs:
            - type: ParentStreamConfig
              parent_key: encodedKey
              partition_field: parentKey
              stream:
                $ref: "#/definitions/streams/accounts"
              incremental_dependency: true
      incremental_sync:
        type: DatetimeBasedCursor
        cursor_field: lastModifiedDate
        cursor_datetime_formats:
          - "%Y-%m-%dT%H:%M:%S%z"
        datetime_format: "%Y-%m-%dT%H:%M:%S%z"
        start_datetime:
          type: MinMaxDatetime
          datetime: "{{ config[\"start_date\"] }}"
          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
        global_substream_cursor: true
      schema_loader:
        type: InlineSchemaLoader
        schema:
          $ref: "#/schemas/loan_schedule"
does that seem right?
k
Your config looks largely correct! Here are a few observations: Looks Good ✅ • 
incremental_dependency: true
on the parent stream config — correct placement and will ensure only updated parent records generate partitions. • 
global_substream_cursor: true
on the 
incremental_sync
— correct for a child stream with no meaningful child-level cursor. • 
cursor_datetime_formats
with 
%Y-%m-%dT%H:%M:%S%z
— correct for the format we identified earlier. • 
DpathExtractor
with 
field_path: [installments]
— correct if the schedule response wraps records in an 
installments
array. One Thing to Check ⚠️ The 
cursor_field: lastModifiedDate
— since this is the schedule/installment endpoint, you need to confirm that the individual records extracted from the 
installments
array actually contain a 
lastModifiedDate
field. If installment records don't have this field, the cursor won't advance correctly. If installment records don't have their own 
lastModifiedDate
, a common workaround is to use a transformation to inject the parent's cursor value onto each child record:
Copy code
transformations:
  - type: AddFields
    fields:
      - type: AddedFieldDefinition
        path:
          - lastModifiedDate
        value: "{{ stream_partition.parentKey }}"  # or a parent record field if available
Though the knowledge sources don't explicitly confirm this pattern for your exact case, so treat this with some caution. [Incremental sync docs] Summary The overall structure is correct. The main thing to verify before testing is whether 
lastModifiedDate
actually exists on the extracted 
installments
records from the API response. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai i need this transformation to use the value of the lastModifiedDate field in the parent record, not the parentKey
k
To inject the parent record's 
lastModifiedDate
value onto each child installment record, you need to use 
extra_fields
in the 
ParentStreamConfig
to pass it through the stream slice, then reference it via 
stream_slice.extra_fields
in the transformation. Update your config like this:
Copy code
partition_router:
  type: SubstreamPartitionRouter
  parent_stream_configs:
    - type: ParentStreamConfig
      parent_key: encodedKey
      partition_field: parentKey
      stream:
        $ref: "#/definitions/streams/accounts"
      incremental_dependency: true
      extra_fields:
        - ["lastModifiedDate"]   # pull this field from the parent record into the slice
Then in your transformations:
Copy code
transformations:
  - type: AddFields
    fields:
      - type: AddedFieldDefinition
        path:
          - lastModifiedDate
        value: "{{ stream_slice.extra_fields['lastModifiedDate'] }}"
This pattern is used in the Stripe connector, for example with the 
updated
field on `payout_balance_transactions`: [Stripe manifest]
Copy code
extra_fields:
  - ["updated"]
# ...
transformations:
  - type: AddFields
    fields:
      - path:
          - updated
        value: "{{ stream_slice.extra_fields['updated'] }}"
        value_type: integer
The 
extra_fields
property is also documented in the 
ParentStreamConfig
reference as: "Array of field paths to include as additional fields in the stream slice... Accessible via 
stream_slice.extra_fields
. Missing fields are set to 
None
." [ParentStreamConfig reference] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai it seems to be accepting the change but when I hit test, there is no lastModifiedDate present in the output records
k
This is a known issue worth investigating from a few angles. The knowledge sources highlight some relevant points: Things to Check 1. Verify extra_fields is actually populated The 
ParentStreamConfig
documentation notes that "Missing fields are set to `None`". [ParentStreamConfig reference] So first, temporarily add a transformation to output the entire 
extra_fields
object to confirm it's being populated at all:
Copy code
transformations:
  - type: AddFields
    fields:
      - path:
          - debug_extra_fields
        value: "{{ stream_slice.extra_fields }}"
If 
debug_extra_fields
shows 
None
or 
{}
, then 
lastModifiedDate
is not being found on the parent record — check that the field name matches exactly (case-sensitive) what the parent 
accounts
stream returns. 2. Check if the field is nested in the parent record 
extra_fields
paths are arrays to support nested access. If 
lastModifiedDate
is nested inside the parent record (e.g. under a sub-object), you'd need to specify the full path:
Copy code
extra_fields:
  - ["someObject", "lastModifiedDate"]
3. Known bug with stream_partition in transformations There is a known bug reported where 
stream_partition
values are not correctly resolved inside transformations. [GitHub issue] While this was reported for 
stream_partition
directly rather than 
stream_slice.extra_fields
, it's worth noting as a potential related issue depending on your Airbyte version. 4. Check your schema Make sure 
lastModifiedDate
is defined in your 
loan_schedule
schema (
#/schemas/loan_schedule
), otherwise it may be stripped from output records even if the transformation adds it correctly. Debugging Steps in Order 1. Add the 
debug_extra_fields
transformation above and re-test to confirm the value is present in the slice. 2. Verify the exact field name on the parent 
accounts
stream records. 3. Confirm 
lastModifiedDate
is in your inline schema. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai i get this message in the logs section of the test screen: Path [] does not have field 
lastModifiedDate
in the schema and hence can't be marked as required.
k
The log message 
Path [] does not have field 'lastModifiedDate' in the schema and hence can't be marked as required
is telling you that 
lastModifiedDate
is missing from your 
loan_schedule
schema definition. This confirms what was mentioned in the previous debugging step — even if the 
AddFields
transformation is correctly injecting the value, Airbyte validates fields against the schema and the field won't appear in output records if it isn't declared there. Fix Add 
lastModifiedDate
to your 
#/schemas/loan_schedule
inline schema. Based on the datetime format we identified earlier (
%Y-%m-%dT%H:%M:%S%z
), it should be declared as a string with a datetime format:
Copy code
loan_schedule:
  type: object
  $schema: http://json-schema.org/draft-07/schema#
  additionalProperties: true
  properties:
    encodedKey:
      type:
        - string
    lastModifiedDate:
      type:
        - "null"
        - string
      format: date-time
      airbyte_type: timestamp_with_timezone
    # ... rest of your existing fields
The key points: • Allow 
null
as a type since 
extra_fields
sets missing fields to 
None
per the docs. [ParentStreamConfig reference] • Use 
format: date-time
and 
airbyte_type: timestamp_with_timezone
to match the ISO 8601 format with UTC offset that this field contains. After adding this to the schema, re-run the test and 
lastModifiedDate
should appear in the output records. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai when i added this transformation:
Copy code
transformations:
  - type: AddFields
    fields:
      - path:
          - debug_extra_fields
        value: "{{ stream_slice.extra_fields }}"
no field called debug_extra_fields appeared in the output
k
This is a more fundamental problem — if even a static debug field isn't appearing, the issue is likely not with 
extra_fields
specifically but with the transformations not being applied at all to the output records. The knowledge sources do not explicitly document why this would happen, but based on what they do contain, here are the most likely causes: Things to Check 1. Verify debug_extra_fields is in your schema As we just established, fields not declared in the schema won't appear in output records. Make sure you temporarily add it to your schema:
Copy code
debug_extra_fields:
  type:
    - "null"
    - string
2. Try a fully static value first To isolate whether transformations are working at all, try the simplest possible case:
Copy code
transformations:
  - type: AddFields
    fields:
      - path:
          - test_field
        value: "hello"
If 
test_field: "hello"
does not appear in output records, then transformations are not being applied to this stream at all, which points to a configuration issue with where 
transformations
is placed in your YAML — it must be a direct property of the 
DeclarativeStream
, not nested inside 
retriever
or elsewhere. [Record Selector docs] The correct placement is:
Copy code
loan_schedule:
  type: DeclarativeStream
  name: loan_schedule
  primary_key:
    - encodedKey
  retriever:
    ...
  incremental_sync:
    ...
  transformations:       # must be at this level, sibling to retriever
    - type: AddFields
      fields:
        - path:
            - test_field
          value: "hello"
  schema_loader:
    ...
Can you confirm where 
transformations
is currently placed in your full stream config? 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai here is the stream config:
Copy code
loan_schedule:
      type: DeclarativeStream
      name: loan_schedule
      primary_key:
        - encodedKey
      retriever:
        type: SimpleRetriever
        requester:
          $ref: "#/definitions/base_requester"
          path: /loans/{{ stream_partition.parentKey }}/schedule
          http_method: GET
          request_parameters:
            detailsLevel: FULL
          request_headers:
            Accept: application/vnd.mambu.v2+json
        record_selector:
          type: RecordSelector
          extractor:
            type: DpathExtractor
            field_path:
              - installments
        partition_router:
          type: SubstreamPartitionRouter
          parent_stream_configs:
            - type: ParentStreamConfig
              parent_key: encodedKey
              partition_field: parentKey
              stream:
                $ref: "#/definitions/streams/accounts"
              incremental_dependency: true
              extra_fields:
                - - lastModifiedDate
      incremental_sync:
        type: DatetimeBasedCursor
        cursor_field: lastModifiedDate
        cursor_datetime_formats:
          - "%Y-%m-%dT%H:%M:%S%z"
        datetime_format: "%Y-%m-%d"
        start_datetime:
          type: MinMaxDatetime
          datetime: "{{ config[\"start_date\"] }}"
          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
        global_substream_cursor: true
      transformations:
        - type: AddFields
          fields:
            - path:
                - parent_key
              value: "{{ stream_slice.parentKey }}"
            - path:
                - debug_extra_fields
              value: "{{ stream_slice.extra_fields }}"
      schema_loader:
        type: InlineSchemaLoader
        schema:
          $ref: "#/schemas/loan_schedule"
The parent_key field in the AddFields transformation DOES appear in the output, but debug_extra_fields doe snot
k
This is very useful information! Since 
parent_key
appears but 
debug_extra_fields
does not, we can confirm that: 1. ✅ Transformations are being applied correctly 2. ✅ The transformation placement is correct 3. ❌ 
stream_slice.extra_fields
is specifically not resolving The most likely cause is that 
lastModifiedDate
is not being found on the parent 
accounts
stream records, and since the 
ParentStreamConfig
docs state "Missing fields are set to `None`", the field is 
None
and not rendering. [ParentStreamConfig reference] Debugging Steps 1. Verify the exact field name on the parent stream Go to your 
accounts
stream in the Connector Builder and run a test. Look at the raw output records and confirm: • Is the field called exactly 
lastModifiedDate
(case-sensitive)? • Or is it something like 
lastModifiedDate
, 
last_modified_date
, 
updatedAt
etc.? 2. Try outputting the full stream_slice to see what's available
Copy code
transformations:
  - type: AddFields
    fields:
      - path:
          - debug_slice
        value: "{{ stream_slice }}"
This will show you the entire slice object including what 
extra_fields
actually contains, which will confirm whether the field is being passed through at all. Once you confirm the exact field name from the parent records, update 
extra_fields
accordingly:
Copy code
extra_fields:
  - ["exactFieldNameHere"]
👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai this is what the debug_slice returns
Copy code
"debug_slice": {
      "parentKey": "8a19af999d8fbc13019d90de87ab46a7",
      "start_time": "2026-08-10",
      "end_time": "2026-08-20"
    }
k
The 
debug_slice
output reveals the problem clearly — 
extra_fields
is not present in the slice at all. The slice only contains 
parentKey
, 
start_time
, and 
end_time
, which means the 
extra_fields
mechanism is not picking up 
lastModifiedDate
from the parent records. This confirms that 
lastModifiedDate
is not a top-level field on the parent 
accounts
stream records as returned, or the field name doesn't match exactly. Next Step Go to your 
accounts
stream in the Connector Builder, run a test, and inspect a raw output record carefully. You are looking for: 1. The exact field name (case-sensitive) for the last modified date 2. Whether it is nested inside a sub-object rather than at the top level If it is nested, for example:
Copy code
{
  "_lastModifiedDate": "2026-08-10T10:26:11+01:00",
  "someObject": {
    "lastModifiedDate": "2026-08-10T10:26:11+01:00"
  }
}
Then your 
extra_fields
path needs to reflect that nesting:
Copy code
extra_fields:
  - ["someObject", "lastModifiedDate"]
The 
ParentStreamConfig
docs confirm that 
extra_fields
paths are arrays of strings to support nested field access. [ParentStreamConfig reference] Once you find the correct field name/path in the parent records, share it here and we can update the config accordingly. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai here is a full raw output record from the accounts stream test:
Copy code
{
  "encodedKey": "8a19af999d8fbc13019d90de87ab46a7",
  "id": "DJBC713",
  "accountHolderType": "CLIENT",
  "accountHolderKey": "8a19af999d8fbc13019d90dec8ad473a",
  "creationDate": "2026-04-15T15:51:51+01:00",
  "approvedDate": "2026-04-15T15:51:59+01:00",
  "lastModifiedDate": "2026-08-10T10:26:11+01:00",
  "activationTransactionKey": "8a19dab89d8fb261019d91a0b6287476",
  "closedDate": "2026-06-15T15:18:10+01:00",
  "lastAccountAppraisalDate": "2026-06-15T00:54:32+01:00",
  "accountState": "CLOSED",
  "accountSubState": "REPAID",
  "productTypeKey": "8a1924ca9d66fd3e019d677f02d80a1c",
  "loanName": "Unsecured Lending",
  "loanAmount": 120000,
  "paymentMethod": "HORIZONTAL",
  "accruedInterest": 0,
  "interestFromArrearsAccrued": 0,
  "lastInterestAppliedDate": "2026-06-15T15:17:59+01:00",
  "accruedPenalty": 0,
  "allowOffset": false,
  "arrearsTolerancePeriod": 0,
  "accountArrearsSettings": {
    "encodedKey": "8a19af999d8fbc13019d919bd21e62a1",
    "toleranceCalculationMethod": "ARREARS_TOLERANCE_PERIOD",
    "dateCalculationMethod": "ACCOUNT_FIRST_WENT_TO_ARREARS",
    "nonWorkingDaysMethod": "INCLUDED",
    "tolerancePeriod": 0
  },
  "latePaymentsRecalculationMethod": "OVERDUE_INSTALLMENTS_INCREASE",
  "balances": {
    "redrawBalance": 0,
    "principalDue": 0,
    "principalPaid": 120000,
    "principalBalance": 0,
    "interestDue": 0,
    "interestPaid": 1843.96,
    "interestBalance": 0,
    "interestFromArrearsBalance": 0,
    "interestFromArrearsDue": 0,
    "interestFromArrearsPaid": 27.87,
    "feesDue": 0,
    "feesPaid": 0,
    "feesBalance": 0,
    "penaltyDue": 0,
    "penaltyPaid": 0,
    "penaltyBalance": 0,
    "holdBalance": 0
  },
  "disbursementDetails": {
    "encodedKey": "8a19af999d8fbc13019d919bd21e62a0",
    "expectedDisbursementDate": "2026-03-11T00:00:00Z",
    "disbursementDate": "2026-03-11T00:00:00Z",
    "firstRepaymentDate": "2026-04-11T00:00:00+01:00",
    "transactionDetails": {
      "encodedKey": "8a19af999d8fbc13019d90de87ab46a8",
      "transactionChannelKey": "8a19596d9d369739019d3eea893435b3",
      "transactionChannelId": "DISB-US",
      "internalTransfer": false
    }
  },
  "prepaymentSettings": {
    "prepaymentRecalculationMethod": "REDUCE_AMOUNT_PER_INSTALLMENT",
    "principalPaidInstallmentStatus": "PAID",
    "applyInterestOnPrepaymentMethod": "AUTOMATIC"
  },
  "penaltySettings": {
    "loanPenaltyCalculationMethod": "NONE"
  },
  "scheduleSettings": {
    "hasCustomSchedule": true,
    "principalRepaymentInterval": 1,
    "gracePeriod": 0,
    "gracePeriodType": "NONE",
    "repaymentInstallments": 24,
    "shortMonthHandlingMethod": "LAST_DAY_IN_MONTH",
    "fixedDaysOfMonth": [
      11
    ],
    "repaymentPeriodCount": 0,
    "scheduleDueDatesMethod": "FIXED_DAYS_OF_MONTH",
    "periodicPayment": 0,
    "repaymentPeriodUnit": "DAYS",
    "repaymentScheduleMethod": "DYNAMIC"
  },
  "interestSettings": {
    "interestRateSource": "FIXED_INTEREST_RATE",
    "accrueInterestAfterMaturity": false,
    "interestApplicationMethod": "REPAYMENT_DUE_DATE",
    "interestBalanceCalculationMethod": "ONLY_PRINCIPAL",
    "interestCalculationMethod": "DECLINING_BALANCE_DISCOUNTED",
    "interestChargeFrequency": "ANNUALIZED",
    "interestRate": 6,
    "interestType": "SIMPLE_INTEREST",
    "accrueLateInterest": true
  },
  "futurePaymentsAcceptance": "NO_FUTURE_PAYMENTS",
  "currency": {
    "currencyCode": "GBP",
    "code": "GBP"
  },
  "allowAdvanceRepayment": false,
  "_selinaCollectionBankAccount": {
    "collectionAccountNumber": "987665433",
    "collectionSortCode": "11-22-34",
    "collectionAccountName": "Collection Account"
  },
  "_selinaApplication": {
    "reference": "US123456",
    "appId": "hello1234"
  },
  "_selinaCustomerBankAccount": {
    "customerSortCode": "11-22-33",
    "customerAccountHolderName": "Elle Woods",
    "customerAccountNumber": "123456"
  }
}
k
Good news — 
lastModifiedDate
is a top-level field on the parent record, so the field name and path are correct. The issue must be elsewhere. Looking at the raw record, I notice that your 
accounts
stream likely has transformations applied that restructure the raw API response into this clean output. The key question is: does 
lastModifiedDate
exist at the top level of the raw API response, or is it added by a transformation? 
extra_fields
pulls fields from the raw parent record before transformations are applied. So if 
lastModifiedDate
is added to the accounts stream via an 
AddFields
transformation, it won't be available to 
extra_fields
. Check Your accounts Stream Config Look at your 
accounts
stream definition and check: 1. Is 
lastModifiedDate
returned directly by the API, or is it added via a transformation like:
Copy code
transformations:
  - type: AddFields
    fields:
      - path:
          - lastModifiedDate
        value: '{{ record["lastModifiedDate"] }}'
2. Is there a 
DpathExtractor
with a 
field_path
that unwraps the record from a nested object? If so, 
lastModifiedDate
may live at a different path in the raw response. Workaround if Transformations are the Issue If 
lastModifiedDate
is indeed added by a transformation on the parent stream, the knowledge sources do not explicitly document a solution for this. The safest workaround would be to not rely on 
extra_fields
 and instead make a note of the parent's cursor value differently — for example, by checking whether the raw API response for the 
/loans/{id}/schedule
endpoint itself contains a 
lastModifiedDate
field on the installment records directly. Can you share what the raw API response looks like for the 
/loans/{id}/schedule
endpoint? That would help determine if the field is available on the child records themselves, removing the need for 
extra_fields
entirely. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai there are no transformations on the accounts stream. it is part of the raw response:
Copy code
{
  "status": 200,
  "body": [
    {
      "encodedKey": "8a19af999d8fbc13019d90de87ab46a7",
      "id": "DJBC713",
      "accountHolderType": "CLIENT",
      "accountHolderKey": "8a19af999d8fbc13019d90dec8ad473a",
      "creationDate": "2026-04-15T15:51:51+01:00",
      "approvedDate": "2026-04-15T15:51:59+01:00",
      "lastModifiedDate": "2026-08-10T10:26:11+01:00",
      "activationTransactionKey": "8a19dab89d8fb261019d91a0b6287476",
      "closedDate": "2026-06-15T15:18:10+01:00",
      "lastAccountAppraisalDate": "2026-06-15T00:54:32+01:00",
      "accountState": "CLOSED",
      "accountSubState": "REPAID",
      "productTypeKey": "8a1924ca9d66fd3e019d677f02d80a1c",
      "loanName": "Unsecured Lending",
      "loanAmount": 120000,
      "paymentMethod": "HORIZONTAL",
      "notes": "",
      "accruedInterest": 0,
      "interestFromArrearsAccrued": 0,
      "lastInterestAppliedDate": "2026-06-15T15:17:59+01:00",
      "accruedPenalty": 0,
      "allowOffset": false,
      "arrearsTolerancePeriod": 0,
      "accountArrearsSettings": {
        "encodedKey": "8a19af999d8fbc13019d919bd21e62a1",
        "toleranceCalculationMethod": "ARREARS_TOLERANCE_PERIOD",
        "dateCalculationMethod": "ACCOUNT_FIRST_WENT_TO_ARREARS",
        "nonWorkingDaysMethod": "INCLUDED",
        "tolerancePeriod": 0
      },
      "latePaymentsRecalculationMethod": "OVERDUE_INSTALLMENTS_INCREASE",
      "balances": {
        "redrawBalance": 0,
        "principalDue": 0,
        "principalPaid": 120000,
        "principalBalance": 0,
        "interestDue": 0,
        "interestPaid": 1843.96,
        "interestBalance": 0,
        "interestFromArrearsBalance": 0,
        "interestFromArrearsDue": 0,
        "interestFromArrearsPaid": 27.87,
        "feesDue": 0,
        "feesPaid": 0,
        "feesBalance": 0,
        "penaltyDue": 0,
        "penaltyPaid": 0,
        "penaltyBalance": 0,
        "holdBalance": 0
      },
      "disbursementDetails": {
        "encodedKey": "8a19af999d8fbc13019d919bd21e62a0",
        "expectedDisbursementDate": "2026-03-11T00:00:00Z",
        "disbursementDate": "2026-03-11T00:00:00Z",
        "firstRepaymentDate": "2026-04-11T00:00:00+01:00",
        "transactionDetails": {
          "encodedKey": "8a19af999d8fbc13019d90de87ab46a8",
          "transactionChannelKey": "8a19596d9d369739019d3eea893435b3",
          "transactionChannelId": "DISB-US",
          "internalTransfer": false
        },
        "fees": []
      },
      "prepaymentSettings": {
        "prepaymentRecalculationMethod": "REDUCE_AMOUNT_PER_INSTALLMENT",
        "principalPaidInstallmentStatus": "PAID",
        "applyInterestOnPrepaymentMethod": "AUTOMATIC"
      },
      "penaltySettings": {
        "loanPenaltyCalculationMethod": "NONE"
      },
      "scheduleSettings": {
        "hasCustomSchedule": true,
        "principalRepaymentInterval": 1,
        "gracePeriod": 0,
        "gracePeriodType": "NONE",
        "repaymentInstallments": 24,
        "shortMonthHandlingMethod": "LAST_DAY_IN_MONTH",
        "fixedDaysOfMonth": [
          11
        ],
        "repaymentPeriodCount": 0,
        "scheduleDueDatesMethod": "FIXED_DAYS_OF_MONTH",
        "periodicPayment": 0,
        "repaymentPeriodUnit": "DAYS",
        "repaymentScheduleMethod": "DYNAMIC",
        "paymentPlan": []
      },
      "interestSettings": {
        "interestRateSource": "FIXED_INTEREST_RATE",
        "accrueInterestAfterMaturity": false,
        "interestApplicationMethod": "REPAYMENT_DUE_DATE",
        "interestBalanceCalculationMethod": "ONLY_PRINCIPAL",
        "interestCalculationMethod": "DECLINING_BALANCE_DISCOUNTED",
        "interestChargeFrequency": "ANNUALIZED",
        "interestRate": 6,
        "interestType": "SIMPLE_INTEREST",
        "accrueLateInterest": true
      },
      "feesSettings": {},
      "assets": [],
      "guarantors": [],
      "fundingSources": [],
      "tranches": [],
      "futurePaymentsAcceptance": "NO_FUTURE_PAYMENTS",
      "currency": {
        "currencyCode": "GBP",
        "code": "GBP"
      },
      "plannedInstallmentFees": [],
      "allowAdvanceRepayment": false,
      "_selinaCollectionBankAccount": {
        "collectionAccountNumber": "987665433",
        "collectionSortCode": "11-22-34",
        "collectionAccountName": "Collection Account"
      },
      "_selinaApplication": {
        "reference": "US123456",
        "appId": "hello1234"
      },
      "_selinaCustomerBankAccount": {
        "customerSortCode": "11-22-33",
        "customerAccountHolderName": "Elle Woods",
        "customerAccountNumber": "123456"
      }
    }
k
The raw API response confirms 
lastModifiedDate
is definitely a top-level field in the raw response, so the 
extra_fields
path is correct. This is puzzling. Looking at your 
debug_slice
output again:
Copy code
{
  "parentKey": "8a19af999d8fbc13019d90de87ab46a7",
  "start_time": "2026-08-10",
  "end_time": "2026-08-20"
}
Notice there is no 
extra_fields
key at all in the slice. This suggests the 
extra_fields
feature may not be functioning as expected in your version of the CDK. Check Your accounts Stream's record_selector The most likely remaining cause is your 
accounts
stream's 
DpathExtractor
. Looking at the raw response, the records are wrapped in an array at the root:
Copy code
{
  "status": 200,
  "body": [{ ... }]
}
Can you check what 
field_path
your 
accounts
stream's extractor uses? If it is extracting from a nested path like 
["body"]
, the 
extra_fields
mechanism should still work on the extracted record. But if there is any mismatch, it could explain why the field isn't found. Pragmatic Workaround Since 
stream_slice.parentKey
is working correctly, and 
parentKey
is the 
encodedKey
of the parent record, you could work around the 
extra_fields
issue entirely by making a separate API call to fetch the parent's 
lastModifiedDate
— but that would be inefficient. A simpler workaround given the knowledge sources don't explain why 
extra_fields
isn't populating, is to raise this as a bug on the Airbyte GitHub with your manifest and the 
debug_slice
output as evidence, since the expected behaviour per the docs is that 
extra_fields
should appear under 
stream_slice.extra_fields
. [ParentStreamConfig reference] In the meantime, could you share your 
accounts
stream config, specifically the 
record_selector
section? That would help narrow down whether the extractor path is causing 
extra_fields
to fail to locate the field. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai here is the accounts config:
Copy code
accounts:
      type: DeclarativeStream
      name: accounts
      primary_key:
        - encodedKey
      retriever:
        type: SimpleRetriever
        requester:
          $ref: "#/definitions/base_requester"
          path: /loans%3Asearch
          http_method: POST
          request_parameters:
            detailsLevel: FULL
          request_headers:
            Accept: application/vnd.mambu.v2+json
            Content-Type: application/json
          request_body_json:
            filterCriteria:
              - field: lastModifiedDate
                value: "{{ stream_interval.start_time }}"
                operator: AFTER_INCLUSIVE
            sortingCriteria:
              field: lastModifiedDate
              order: ASC
        record_selector:
          type: RecordSelector
          extractor:
            type: DpathExtractor
            field_path: []
      incremental_sync:
        type: DatetimeBasedCursor
        cursor_field: lastModifiedDate
        cursor_datetime_formats:
          - "%Y-%m-%dT%H:%M:%S%z"
        datetime_format: "%Y-%m-%d"
        start_datetime:
          type: MinMaxDatetime
          datetime: "{{ config[\"start_date\"] }}"
          datetime_format: "%Y-%m-%dT%H:%M:%SZ"
      schema_loader:
        type: InlineSchemaLoader
        schema:
          $ref: "#/schemas/accounts"
k
The accounts config reveals the likely cause. Your 
accounts
stream uses:
Copy code
field_path: []
This means the extractor treats the entire response body as the record array with no unwrapping. This is fine for record extraction, but it may be the reason 
extra_fields
isn't working — the CDK may not be correctly locating fields when 
field_path
is empty (
[]
). However, I want to be honest: the knowledge sources do not explicitly document how 
extra_fields
interacts with an empty 
field_path
, so I cannot say this with full certainty. Most Pragmatic Workaround Since you can't get 
extra_fields
working right now, and given that 
stream_slice.parentKey
does work, you could use the 
parentKey
(which is the 
encodedKey
) as a proxy cursor for the child stream instead of 
lastModifiedDate
. The logic being: if a parent record appeared in the incremental sync, it was modified after the last cursor, so its schedule needs refreshing. Update your 
loan_schedule
incremental sync to use 
encodedKey
as the cursor field:
Copy code
incremental_sync:
  type: DatetimeBasedCursor
  cursor_field: lastModifiedDate
  cursor_datetime_formats:
    - "%Y-%m-%dT%H:%M:%S%z"
  datetime_format: "%Y-%m-%dT%H:%M:%S%z"
  start_datetime:
    type: MinMaxDatetime
    datetime: "{{ config[\"start_date\"] }}"
    datetime_format: "%Y-%m-%dT%H:%M:%SZ"
  global_substream_cursor: true
And inject the parent's 
lastModifiedDate
via the transformation using 
stream_slice.parentKey
to look it up — except that's circular. Recommended Next Step Given the uncertainty, I'd suggest raising this as a bug on the Airbyte CDK GitHub with: • Your 
loan_schedule
stream config • Your 
accounts
stream config • The 
debug_slice
output showing 
extra_fields
is absent The 
extra_fields
feature should work per the docs [ParentStreamConfig reference], and the empty 
field_path: []
interaction is the most suspicious difference between your setup and the expected behaviour. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
s
@kapa.ai do you know what version of airbyte the extra_fields functionality is supported on? perhaps my deployment needs updating
k
I'm sorry, but the knowledge sources do not contain enough information to confirm which specific Airbyte version first introduced 
extra_fields
support, so I cannot give you a reliable version number to check against. What I can suggest is: 1. Check the Airbyte CDK changelog at github.com/airbytehq/airbyte-python-cdk/releases and search for 
extra_fields
to find when it was introduced. 2. Check your current CDK version — in the Connector Builder, the manifest has a 
version
field at the top. You can compare this against the CDK changelog to see if 
extra_fields
was available at that version. 3. Check your Airbyte platform version — if you are self-hosting with 
abctl
, run:abctl local statusto see your current platform version, then check the Airbyte releases page to see if an update is available. Given that your 
debug_slice
shows no 
extra_fields
key at all rather than a 
None
value, a version gap is a plausible explanation. Updating both the platform and the CDK version declared in your manifest would be a reasonable thing to try. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
3 Views
Open in Slack
PreviousNext
Powered by