Does anyone have any troubleshooting tips or workarounds for the Redshift destination connector problem where any incoming VARCHAR field over 65535 blows up the data sync? I’ve tried fumbling around and truncating long data from VARCHAR columns but I haven’t been able to solve it that way. I see messages like these in the worker logs but they don’t mean much to me (I can’t tell where it’s getting stuck):

13 of 33 ERROR creating incremental model loading.log_scd................................................... [ERROR in 16.07s]
Database Error in model log_scd (models/generated/airbyte_incremental/scd/loading/log_scd.sql)
  Invalid input
  DETAIL:  
    -----------------------------------------------
    error:  Invalid input
    code:      8001
    context:   CONCAT() result too long for type varchar(65535)
    query:     417552
    location:  string_ops.cpp:108
    process:   query0_113_417552 [pid=690]
    -----------------------------------------------
  compiled SQL at ../build/run/airbyte_utils/models/generated/airbyte_incremental/scd/loading/log_scd.sql

Any ideas would be greatly appreciated! Here is the root issue in GitHub: https://github.com/airbytehq/airbyte/issues/14441

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2817 to ensure timely resolution!

Hey there, if I'm understanding correctly it looks like the root issue is caused by the normalization step. There is some discussion on how this problem can be tackled in here: https://github.com/airbytehq/airbyte/issues/16412#issuecomment-1281613116 If I find anything more I will definitely let you know.

Thanks for the reply! I actually had a comment in that thread already and just added a new one. The workaround that we used to address the problem was to create a new view that is just a subset of the fields in the source table that was failing the hash.

Interesting, thanks very much for sharing! I'm glad to hear you found a workaround.

Yep, we’re good. I think it’ll work OK for us but hoping that a permanent solution isn’t too far away. Definitely hoping the Redshift Destination connector goes GA this quarter. We’re starting to use it with production customers starting in January.

Hi @Sunny Hashmi (Airbyte) I also got hit with this issue. What airbyte version that does break this function? I have old instance running 0.30.xx and it works. Now with latest instance 0.40.xx, it stops working.

Hey @Eric Hartono, it looks like there has been movement on the related github issues since your post - please feel free to share any information you have on those issues to help with resolution

Nothing new to add @Sunny Hashmi (Airbyte). Column selection would be a great workaround, and also a great feature to have.