Hi Team
I have loaded Zendesk data to s3 in parque...
# troubleshootings
Srinivasula Reddy Yennapusa
03/01/2022, 6:19 AMHi Team
I have loaded Zendesk data to s3 in parquet file format.
Note: the data contains json columns.
Now, while I am trying to copy this s3 parquet file to redshift, I am getting the below issue:
SQL Error [XX000]: ERROR: Spectrum Scan Error
Detail:
-----------------------------------------------
error: Spectrum Scan Error
code: 15007
context: Unsupported implicit cast: Column s3://airbyte-sync-tn/data_sync/test_zendesk_super/tickets/2022_02_28_1646039367500_0.parquet.via, FromType: struct<struct<struct<byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,byte_array,map<b
Srinivasula Reddy Yennapusa
03/01/2022, 6:19 AMSo here via column is json and in redshift the corresponding column is in *VARCHAR*(65535).
u
user
03/01/2022, 6:21 AMAlso when I have changed *VARCHAR*(65535) to SUPER data type for via column in redshift. I am getting getting the below error :
SQL Error [XX000]: ERROR: External Catalog Error.
Detail:
-----------------------------------------------
error: External Catalog Error.
code: 16000
context: Unsupported column type found for column: 4. Remove the column from the projection to continue.
i
Ivan Coyoca
03/01/2022, 7:57 AMHey can you share the complete log of the sync?
a
Adam Laurenčík
03/01/2022, 8:39 AMthe above is the error message, no log as such
Adam Laurenčík
03/01/2022, 10:02 AM@Harshith (Airbyte) can you please let me know
Adam Laurenčík
03/01/2022, 10:14 AMThis is basically problem on changing the schema on destination right?
Adam Laurenčík
03/01/2022, 10:20 AMFrom the error message, it looks like the “via” column (json data) is unable to copy in redshift either in VARCHAR or SUPER datatype.
How we should copy this? @Harshith (Airbyte)
Adam Laurenčík
03/01/2022, 10:21 AMCan you share the catalog for the connection?
m
Mahesh
03/01/2022, 10:23 AMAre you referring to this?
Mahesh
03/01/2022, 10:23 AMAfter creating destination you will be creating a connection right where you will choose streams that one
Mahesh
03/01/2022, 10:27 AMQuick question? Is it possible to transfer data directly to Redshift suing Airbyte rather than loading file from S3
Mahesh
03/01/2022, 10:29 AMYes, but that consumes a lot of Redshift cpu capacity.
3 Views