I am trying to build a parent - child stream in th...
# ask-ai
l
I am trying to build a parent - child stream in the connector builder and am getting this error -- Client error : 400 Bad Request {"exceptionStack":"Traceback (most recent call last):\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/connector_builder_handler.py\", line 66, in read_stream\n stream_read = handler.get_message_groups(source, config, configured_catalog, state, limits.max_records)\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/connector_builder/message_grouper.py\", line 125, in get_message_groups\n schema = schema_inferrer.get_stream_schema(configured_stream.stream.name)\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/utils/schema_inferrer.py\", line 209, in get_stream_schema\n self._add_required_properties(self._clean(self.stream_to_builder[stream_name].to_schema()))\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/utils/schema_inferrer.py\", line 107, in _clean\n self._clean(value)\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/utils/schema_inferrer.py\", line 109, in _clean\n self._clean(node[\"items\"])\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/utils/schema_inferrer.py\", line 107, in _clean\n self._clean(value)\n File \"/home/airbyte/.pyenv/versions/3.9.11/lib/python3.9/site-packages/airbyte_cdk/utils/schema_inferrer.py\", line 112, in _clean\n if isinstance(node[\"type\"], list)\nKeyError 'type'\n","exceptionClassName":"io.airbyte.protocol.models.AirbyteTraceMessage","message":"Error reading stream with config=
k
It looks like you're encountering a
400 Bad Request
error while trying to build a parent-child stream in the connector builder. The error trace indicates a
KeyError: 'type'
in the
schema_inferrer.py
file. This suggests that there might be an issue with the schema definition for one of your streams. Here are a few steps you can take to troubleshoot and resolve this issue: 1. Check Schema Definitions: Ensure that all your stream schemas are correctly defined and include the
type
field. The error indicates that the
type
field is missing in one of the schema nodes. 2. Validate JSON Schema: Use a JSON schema validator to ensure that your schema definitions are correct and adhere to the expected format. 3. Simplify and Isolate: Try isolating the problematic stream by simplifying your configuration. Remove other streams and test each one individually to identify which stream's schema is causing the issue. 4. Review Parent-Child Configuration: Ensure that the parent-child relationship is correctly configured. The child stream should correctly reference the parent stream's field in its URL path. 5. Refer to Similar Issues: You can refer to similar issues that have been reported and resolved. For example, [Issue #33738](https://github.com/airbytehq/airbyte/issues/33738) was closed as a duplicate of a feature discussion, which might provide additional context or solutions. If you continue to face issues, you might want to open a new issue on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues) with detailed information about your configuration and the error trace. For more detailed guidance, you can also refer to the [Airbyte documentation](https://docs.airbyte.com/) or seek help from the community on the [Airbyte forum](https://discuss.airbyte.io/). If you have any specific parts of your schema or configuration that you would like to share, feel free to do so, and I can help you review them for potential issues.
l
@kapa.ai I just figure out that the parent id is coming through as a decimal value when it is set to number or string in the parent declared schema.
k
It sounds like you're encountering an issue where the parent ID is being returned as a decimal value, even though it is set to be either a number or a string in the parent declared schema. This can be addressed by ensuring that the schema accurately reflects the data types that the API might return. Here are a few steps you can take to resolve this issue: 1. Update the Declared Schema: Ensure that the declared schema allows for both number and string types. This can be done by updating the schema to include both types. 2. Enable Casting: Use the
Cast Record Fields to Schema Types
feature to ensure that the fields are cast to the desired type during processing. ### Example Steps: 1. Update the Declared Schema: - In the Declared Schema tab, disable
Automatically import detected schema
. - Update the schema to allow both number and string types for the parent ID.
Copy code
json     {       "parent_id": {         "type": ["number", "string"]       }     }
2. Enable Casting: - Enable
Cast Record Fields to Schema Types
in the Record Selector component. ### Example Configuration:
Copy code
json {   "streams": [     {       "name": "example_stream",       "schema": {         "properties": {           "parent_id": {             "type": ["number", "string"]           }         }       },       "record_selector": {         "type": "RecordSelector",         "extractor": {           "type": "DpathExtractor",           "field_path": []         },         "cast_record_fields_to_schema_types": true       }     }   ] }
By following these steps, you ensure that the
parent_id
field is correctly interpreted as either a number or a string, depending on the data returned by the API. For more detailed information, you can refer to the [Record Processing Documentation](https://docs.airbyte.com/connector-development/connector-builder-ui/record-processing#cast-record-fields-to-schema-types). If you continue to experience issues, please provide more details or check the [Airbyte GitHub Issues](https://github.com/airbytehq/airbyte/issues) for similar cases.
l
@kapa.ai it is still sending the parent id as a decimal value when I have the type set to number or string. and I have the cast_record_fields_to_schema_types set to ture
k
A message from kapa.ai
If the
parent_id
is still being sent as a decimal value despite setting the type to
number
or
string
and enabling
cast_record_fields_to_schema_types
, it might be due to how the casting is being handled internally. Here are a few steps to troubleshoot and potentially resolve this issue: 1. Check the Declared Schema: Ensure that the declared schema is correctly set to allow both
number
and
string
types for the
parent_id
. 2. Review the Casting Logic: Verify that the
cast_record_fields_to_schema_types
feature is correctly enabled and functioning as expected. ### Example Configuration: 1. Update the Declared Schema: - Ensure the schema allows both
number
and
string
types for the
parent_id
.
Copy code
json     {       "parent_id": {         "type": ["number", "string"]       }     }
2. Enable Casting: - Ensure
cast_record_fields_to_schema_types
is set to
true
.
Copy code
json     {       "streams": [         {           "name": "example_stream",           "schema": {             "properties": {               "parent_id": {                 "type": ["number", "string"]               }             }           },           "record_selector": {             "type": "RecordSelector",             "extractor": {               "type": "DpathExtractor",               "field_path": []             },             "cast_record_fields_to_schema_types": true           }         }       ]     }
### Additional Steps: 1. Review the API Response: Ensure that the API response is consistent and does not contain unexpected data types that might interfere with the casting process. 2. Check for Known Issues: Review the [Airbyte GitHub Issues](https://github.com/airbytehq/airbyte/issues) for any known issues related to casting or schema handling. For example, there was a related issue with bad casting in the
_cast_record_fields_if_needed
function ([GitHub Issue #24222](https://github.com/airbytehq/airbyte/issues/24222)). 3. Update Airbyte: Ensure you are using the latest version of Airbyte, as updates may include fixes for casting and schema handling issues. If the issue persists, you might want to open a new issue on the [Airbyte GitHub repository](https://github.com/airbytehq/airbyte/issues) with detailed information about your configuration and the problem you are encountering. This will help the Airbyte team to investigate and provide a solution.