https://linen.dev logo
Join Slack
Powered by
Hi everyone, I am working on a connector using the...
# replication-troubleshooting
t
Hi everyone, I am working on a connector using the low code configuration framework. The API I build the connector for has changing structures depending on the endpoint. I know that the record selector () can be used to specify the field but how can I account for different fields depending on the stream? For example, for the /model api, the response is 
{ meta: {}, status: {}, models:{}}
For /workspaces it is 
{ meta: {}, status: {}, eirkspaces:{}}
And for another type of endpoints with tabular data it is 
{pages: {}, columnCoordinates: {}, rows: {}
How can I account for different response structures per stream? Can I specify a record selector per stream? Thank you for your help!
n
Hi Till, my first thought is that this would be defined in the stream schemas, am I misunderstanding your question? Could you link to the API documentation?
t
Hi @Nataly Merezhuk (Airbyte), maybe there is a misunderstanding on my side but as far as I understood one needs to specify where the actual records are placed in a stream. In the first two examples, the actual records are in models: / workspaces: while meta: and status: are metadata. In the third example, the record starts at the root of the response. While I can specify the respective schema files accordingly,
when running the pytest on the connector with the following schema structure: 
{
"type": "object",
"required": ["workspaces"],
"properties": {
"workspaces": {
"type": "array",
"items": {
"type": "object",
"required": ["id", "name"],
"properties": {
"id": { type": "string"
}, 
"name": { "type": "string" },
<...>
}
}
}
}
}
and this configuration: 
workspaces_stream:
$ref: "*ref(definitions.base_stream)"
$options:
name: "workspaces"
primary_key: "id"
path: "/workspaces"
I get the error thas “id” is not available as PK
Alright, I found a possible solution, in my configuration: 
definitions:
selector:
extractor:
field_pointer: [ "{{ options.extractor_field_pointer  }}" ]
<...>
workspaces_stream:
$ref: "*ref(definitions.base_stream)"
$options:
name: "workspaces"
extractor_field_pointer: "workspaces"
primary_key: "id"
path: "/workspaces"
and then the schemas in a way that assume the field_pointer to be the “root”. Please let me know if there is a smarter / best practice way.
n
This sounds correct!
3 Views
Open in Slack
PreviousNext
Powered by