<@U04SE163WC8> i'm using the CDK UI to develop a n...
# help-connector-development
r
@kapa.ai i'm using the CDK UI to develop a new connector. the value i need to extract for my substream partition router is nested within a JSON object. how can i extract that nested value to use it as my partition router?
👀 1
k
A message from kapa.ai
To extract a nested value for your substream partition router, you can use the SubstreamPartitionRouter component. You'll need to configure the following fields: 1. Parent stream: Define the records of which stream should be used as partitions. 2. Parent key: Specify the property on the parent stream record that should become the partition value (in most cases, this is some form of ID). 3. Current partition value identifier: Choose an identifier for the variable holding the current partition value. You can use this identifier in the path of the stream using the
{{ stream_partition.<identifier> }}
syntax. Here's an example of how to configure the SubstreamPartitionRouter component:
Copy code
yaml partition_router:   type: SubstreamPartitionRouter   parent_stream_configs:     - parent_stream: parent_stream_name       parent_key: nested_key_name   current_partition_value_identifier: chosen_identifier
In this example, replace
parent_stream_name
with the name of the parent stream,
nested_key_name
with the nested key you want to extract, and
chosen_identifier
with the identifier you choose for the current partition value. For more information on the SubstreamPartitionRouter component, refer to the [documentation](https://docs.airbyte.com/connector-development/config-based/understanding-the-yaml-file/reference#SubstreamPartitionRouter).
👀 1
r
@kapa.ai where are you getting the term
current_partition_value_identifier
from? i can't find it referenced in Airbyte's docs
k
A message from kapa.ai
Apologies for the confusion earlier. You are correct that the term
current_partition_value_identifier
is not mentioned in the Airbyte documentation. I made an error in my response. To extract a nested value for your substream partition router, you should configure the "Parent key" field to point to the nested property you want to use as the partition value. You can use dot notation to access nested properties. For example, if your parent stream record looks like this:
Copy code
json {   "id": 123,   "details": {     "nested_id": 456   } }
You can set the "Parent key" field to
details.nested_id
to use the nested value as the partition value. Please refer to the [Substream partition router documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/partitioning#dynamic-and-static-partition-routing) for more information on configuring the partition router.
r
@kapa.ai that's what i tried, and i get the error
Could not perform read with error: Every message grouping should have at least one request and response
✅ 1
k
A message from kapa.ai
New message text here
t
Hey Roberto, did you solve your problem? I’m dealing with the same.. My Parent Key is nested and dot notation not work.
r
hey @Tatiane Corrêa! i had to give up on it 😞. we have a data extractor running off of a python script outside of Airbyte. i was trying to bring that into the Airbyte environment, but it was easier to leave things as-is
🥲 1
t
So..here i’m using Record Selector param to filter the response. The problem was that in the response came pagination params too..so i just filter the field with the real values and it worked out … and i don’t need dot notation anymore
🔥 1