https://linen.dev logo
Join Slack
Powered by
Hello team, I have created custom connector and on...
# help-connector-development
s
Hello team, I have created custom connector and on instance i have uploaded the docker image to get the new connector. Conenctor is working fine in local, and on interface as well. I can see emitted records as well. but when i look into the schema, the main table is blank. What could be the reason ? is this a version issue or am i missing something
m
Did you check the raw table or the final normalized?
s
raw json response is available in raw tables but main table is empty
m
Did you check your schema is correct and matches each record?
s
Yes, schema is correct and its matching as well
m
Can you share the schema and one example of one record?
s
Schema -
api.json
api.json
response -
Copy code
"products": [
        {
            "id": 83047368,
            "created_at": "2019-01-02T15:26:41.000+00:00",
            "updated_at": "2019-08-27T08:26:25.000+00:00",
            "blacklisted": false,
            "average_score": 4.58357,
            "total_reviews": 3554,
            "url": "<https://yotpo.com/go/fk**4Ad>",
            "external_product_id": "yotpo_site_reviews",
            "name": "<http://www>.******.com",
            "description": null,
            "product_specs": [],
            "category": {
                "id": null,
                "name": null
            },
            "images": []
        }
    ]
}
m
You don’t want to ingest the products, you must parse the items inside the array and the schema must match that object.
Copy code
{
  "id": 83047368,
  "created_at": "2019-01-02T15:26:41.000+00:00",
  "updated_at": "2019-08-27T08:26:25.000+00:00",
  "blacklisted": false,
  "average_score": 4.58357,
  "total_reviews": 3554,
  "url": "<https://yotpo.com/go/fk**4Ad>",
  "external_product_id": "yotpo_site_reviews",
  "name": "<http://www>.******.com",
  "description": null,
  "product_specs": [],
  "category": {
    "id": null,
    "name": null
  },
  "images": []
}
You schema should be something like:
Copy code
{
    "$schema": "<http://json-schema.org/draft-07/schema#>",
    "properties": {
       "id": {
              "type": "number"
            },
        "created_at": {
              "type": "number"
            },
     ...
}
s
Okay alright, I'll try this and let you know. Thank you !
@Marcos Marx (Airbyte) Hey it worked properly. Thanks alot ! just one quick question : is it possible to get nested parent nodes in single table ? for e.g: in below imge, we have product_specs, category, images as parent nodes, but i got new table for each node. can we get that in single table ?
Copy code
}
      },
      "required": [
        "id",
        "name"
      ]
    },
    "images": {
      "type": "array",
      "items": [
        {
          "type": "object",
          "properties": {
            "original": {
              "type": "string"
            },
            "square": {
              "type": "string"
            },
            "facebook": {
              "type": "string"
            },
            "facebook_square": {
              "type": "string"
            },
            "kind": {
              "type": "string"
            }
          },
m
for e.g: in below imge, we have product_specs, category, images as parent nodes, but i got new table for each node. can we get that in single table ?
Normalization will do this for you and generate a table like 
parent_images
s
Okay đź‘Ť ty!
@Marcos Marx (Airbyte) One last question, I am almost completed the custom connector setup. I am getting following error while data sync as : 
cursor = column_names[self.cursor_field[0]][0]
2023-04-14 09:53:12 normalization > KeyError: 'since_updated_at'
I am passing following filed in my cursor_filed :
Why I am getting the above error ? could you please help me on it
m
The record you shared with me doesn’t have this field only created_at and updated_at
s
I am giving this filed in my request_parameters. If I keep cursor_paginator as blank then it works. Ill try with updated_at or created_at field then.
2 Views
Open in Slack
PreviousNext
Powered by