Hi All, using Airbyte to move data from aurora pos...
# replication-troubleshooting
k
Hi All, using Airbyte to move data from aurora postgres to redshift using S3 copy strategy. I successfully copied one table with 219 records in 2 and a half minutes (initial load). However, when I try to copy multiple tables (21) the sync job seems to just sit there. Worker log shows
Copy code
source > Max memory limit: 94670422016, JDBC buffer size: 1073741824
Any advice, documentation, blogs that would help tuning airbyte to speed things up?
Copy code
airbyte version 0.40.26 running on EKS
u
I didn't find any issues in Github related to Aurora. Can you share the complete sync log to investigate?
k
Yep, sending your way.
Sent
Hi @Marcos Marx (Airbyte), please let me know if you've had time to review the log and found anything that I need to change. Thank you!
u
The error: 2023-01-26 173841 [42mnormalization[0m > [33mDatabase Error in model query_results (models/generated/airbyte_incremental/airbyte/query_results.sql)[0m2023-01-26 173841 [42mnormalization[0m > Invalid input2023-01-26 173841 [42mnormalization[0m > DETAIL: 2023-01-26 173841 [42mnormalization[0m > -----------------------------------------------2023-01-26 173841 [42mnormalization[0m > error: Invalid input2023-01-26 173841 [42mnormalization[0m > code: 80012023-01-26 173841 [42mnormalization[0m > context: CONCAT() result too long for type varchar(65535)2023-01-26 173841 [42mnormalization[0m > query: 1955778702023-01-26 173841 [42mnormalization[0m > location: string_ops.cpp:1082023-01-26 173841 [42mnormalization[0m > process: query1_168_195577870 [pid=20975]2023-01-26 173841 [42mnormalization[0m > -----------------------------------------------2023-01-26 173841 [42mnormalization[0m > [33mDatabase Error in model scheduled_queries (models/generated/airbyte_incremental/airbyte/scheduled_queries.sql)[0m The Github issue to fix the problem: https://github.com/airbytehq/airbyte/issues/16412 Today there isn't a ETA to fix it :(