Reposting because I think this may have gotten buried over the holiday… Hello! A couple weeks ago we had a very strange occurrence with our sync from mysql 5.6 to bigquery. After a particular “full refresh - overwrite” connection failed to sync 3 consecutive times, subsequent syncs of the same connection began taking double the time to complete. Looking at the logs, the extra time was due to an unexplained gap during normalization. After all the sql scripts were written, there would be an hour of nothing before any of them would start executing. However, eventually dbt would successful complete. Even more strangely, a later sync of the same connection failed 3 consecutive times - afterward subsequent syncs once again took the original amount of time. The sync in question is relatively large (5.73 GB) and copies >200 tables. I’ve included a copy of the logs for one run during the period that there was an hour gap in normalization. We’re on airbyte version 0.40.17 running on GKE. We’re happy the this problem fixed itself but we would like help identifying what happened here and how we can prevent it from happening again. Thanks!! https://files.slack.com/files-pri/T01AB4DDR2N-F04FX3KT9E3/download/0df738fa_6de8_432d_9aa9_a77e48a47a5f_logs_111_txt.txt?origin_team=T01AB4DDR2N

Hello Noah, do you have only this connection or others? The issue is only with this one? Can you check the resources of your cluster (disk and memory?) Sometimes you’re running the sync but all worker slots are in use by others connections. Can you share a previous connection where everything was working and one which took more time to finish?

Hey @Marcos Marx (Airbyte) thanks for the response! We do have other connections but the problem was only with this one - it is by far our largest. The disk and memory in the cluster did look normal. I’m attaching the most recent successful sync of the connection prior the 3 failures the preceded the increase in sync time. I think all the worker slots being in use is a good theory - but it seems weird the issue would only be on this particular connection at the exact point. Thanks for the help!

Are you able to check the pod/container resource for the source/destination? I didn’t find any meaningful information inside the logs could help. Maybe try to duplicate the connection and use less tables to validate the sync can finish using the source and destination