I have been trying to send some data from S3 to Po...
# replication-troubleshootingm
Milind Soni
10/19/2022, 3:38 PMI have been trying to send some data from S3 to Postgres which is in a Jsonl format. I am having a lot of JSON parsing errors specifically "JSON parse error: Column() changed from object to array in row 0" . Is there any option that I can restructure my data so that I can transfer it seamlessly . Please suggest me any steps to achieve the same.
✍️ 1
u
user
10/19/2022, 3:45 PM@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2942 to ensure timely resolution!
s
Saj Dider (Airbyte)
10/19/2022, 4:32 PMHey @Milind Soni, can you share some more info about what versions you’re using for each connector and for airbyte?
Furthermore, can you share some examples of the json data and some of the logs?
Finally, other than restructuring your data, have you considered using something like the Avro format?
m
Milind Soni
10/19/2022, 8:57 PMHi Saj, Thanks for the quick reply. I am sharing a snipped of the data with you
Milind Soni
10/19/2022, 8:58 PM{
"_airbyte_ab_id": "e0esfasfasf96c-4b88-4068-a07d-1c3c08f6c304",
"_airbyte_emitted_at": 1666099658208,
"_airbyte_data": {
"type": "SINGLE_SEND_API",
"counters": {
"dropped": "new"
}
}
},
{
"_airbyte_ab_id": "esdsefsfs3096c-4b88-4068-a07d-1c3c08f6c304",
"_airbyte_emitted_at": 1666099658208,
"_airbyte_data": {
"type": "SINGLE_SEND_API",
"counters": {
"dropped": "new"
}
}
}
Milind Soni
10/19/2022, 9:00 PMVERSION=0.40.3
Milind Soni
10/19/2022, 9:08 PMThanks for the help @Saj Dider (Airbyte)
Milind Soni
10/20/2022, 7:24 AM@Saj Dider (Airbyte) I have converted my data to AVRO format but I keep getting an error of " unhashable type: 'dict'"
s
Saj Dider (Airbyte)
10/20/2022, 2:21 PMHey @Milind Soni, upgrade to the latest versions for everything first and then retry the sync.
Other than that, can you send me some of the raw data from s3? I’m trying to understand how the jsonl data is formatted and why it’s not parsing correctly.
m
Milind Soni
10/20/2022, 2:24 PM@Saj Dider (Airbyte) Okay I explored and found that parquet form might be useful to us. But for nested object I am getting all NULL values after the sync. Can you guide me what should I do. Its not pulling data from the nested objects in the parquet file
s
Saj Dider (Airbyte)
10/21/2022, 4:45 PMHey @Milind Soni.
What version of the s3 connector are you using? There's been a lot of recent work on schema inference and dealing with nested objects. I think converting nested objects into parquet may lead to more issues but I'm not sure.