Hello team! Referring to the <issue on ignoring re...
# feedback-and-requestsa
Arnold Cheong
12/27/2021, 2:29 AMHello team! Referring to the issue on ignoring records too big for Redshift here, what is the recommended solution to address these ignored records? We’d still want them to end up in Redshift eventually, so wondering whether anybody has ideas on how to address this?
Arnold Cheong
12/27/2021, 2:16 PMHi @Arnold Cheong @Chris (deprecated profile) commented the following in the code:
Copy code
Truncate json data instead of throwing whole record away?
or should we upload it into a special rejected record table instead?Arnold Cheong
12/27/2021, 2:17 PMI'd suggest you to open an issue as a feature request on our repo if you think one of this option is valid for your use case, or if you have another solution in mind.
t
Tyler Buth
12/29/2021, 3:18 PMDoes Airbyte have a “dead letter queue” equivalent like there is with Kafka?
p
Paul Cothenet
12/30/2021, 1:54 AMThanks @[DEPRECATED] Augustin Lafanechere, I’ll do that after the holidays. Was just wondering if any other Airbyte Redshift users found workarounds for this as I imagine ignoring records wholesale is hardly a long term solution.
Paul Cothenet
12/30/2021, 2:14 AM@[DEPRECATED] Augustin Lafanechere just to double check, does the existing implementation give any warnings in the logs of any sort if records are ignored?
Paul Cothenet
12/30/2021, 2:14 AM@Chris (deprecated profile) perhaps you’re the better person to answer, please and thank you!
Paul Cothenet
01/03/2022, 5:50 PMa
Alex Malinovskyi
02/13/2022, 12:32 PMUsing Redshift’s SUPER type would handle records of up to 1MB: https://docs.aws.amazon.com/redshift/latest/dg/r_SUPER_type.html
2 Views