*Is this your first time deploying Airbyte*: No De...
# troubleshooting
a
Is this your first time deploying Airbyte: No Deployment Kubernetes Airbyte Version: 0.35.5-alpha Source name/version: Postgres version 0.4.4 ( had to downgrade from most recent one, because of this issue, but the same also happend in 0.4.8) Destination name/version: Postgres version 0.3.14 Step: On sync Description: Hello 👋 I am syncing data between two postgres databases, and the process gets stuck on the “Executing finalization of tables”, it has been like this for the whole night and after that I had to cancel the process. What Is strange is, that when I exclude the biggest table from the list of tables that are being synced ( it has around 20GB in total) everything works correctly… octavia thinking Is there any known issue that airbyte cannot transfer tables bigger than some threshold? Or maybe some way to see more debug messages, to better investigate what is going on? Thank you for any help! octavia thanks (See the attached full logs, as well as the message where the logs got stuck for many hours )
Copy code
2022-02-28 15:58:47 INFO i.a.w.p.a.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-02-28 15:58:47 INFO i.a.i.d.j.JdbcBufferedConsumerFactory(lambda$onCloseFunction$3):178 - Executing finalization of tables.
2022-03-01 07:50:13 INFO i.a.w.t.TemporalAttemptExecution(lambda$getCancellationChecker$3):210 - Running sync worker cancellation...
I also switched to Postgres destination from Mysql (source Mongo) to benefit from the Incremental + Deduped mode. However, I am also running into this same issue after initial sync.
On further exploration I found some queries stuck in locks in Postgres and have tried to remove them as per this https://jaketrent.com/post/find-kill-locks-postgres. Will now run the sync again to see if it helps
m
Hey @Adam Laurenčík , I think the step that get stuck is during the copy of tmp tables to final tables. Could you please check the following: • Are tmp tables filled with the data you expect to replicate • What are the running queries on your destination database during this finalization step? Is the DB idle or the copy is still happening? The good news is that the bottleneck is probably on your destination database if Airbyte was able to successfully deliver the records to tmp tables 🤔