Hi people, doing a sync with source Postgresql and...
# replication-troubleshootingj
Jhon Edison Bambague Calderon
12/14/2022, 2:36 AMHi people, doing a sync with source Postgresql and target Snowflake I'm getting this error: "Sync worker failed" and "airbyte-source gobbler IOException: Connection timed out. Typically happens when cancelling a job", always with the same duration time (4hours) and the same 74GB of data, anyone has happened or something to adjust? Thank you for your recommendations
Jhon Edison Bambague Calderon
12/14/2022, 2:37 AM Copy code
2022-12-13 20:35:07 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):369 - Records read: 11170000 (74 GB)
2022-12-13 20:35:08 INFO i.a.w.g.DefaultReplicationWorker(lambda$getReplicationRunnable$6):369 - Records read: 11171000 (74 GB)
2022-12-13 20:57:29 WARN i.a.c.i.LineGobbler(voidCall):119 - airbyte-source gobbler IOException: Connection timed out. Typically happens when cancelling a job.
2022-12-13 22:47:43 INFO i.a.w.p.KubePodProcess(destroy):664 - (pod: airbyte / source-postgres-read-80-2-fxcsn) - Destroying Kube process.
2022-12-13 22:47:44 destination > Airbyte message consumer: succeeded.Jhon Edison Bambague Calderon
12/14/2022, 2:41 AM Copy code
2022-12-13 22:48:43 ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: java.lang.RuntimeException: java.io.UncheckedIOException: java.net.SocketException: Connection timed out
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:189) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.17.jar:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 143. This warning is normal if the job was cancelled.e
Eli Sigal
12/14/2022, 6:18 AMHi.
it`s a timeout- i.e. it took to much time to pull the data based on the source connectors timeout logic.
not sure it can be fixed other than:
1. create your own connectors that handles this or
2. run different jobs with different data from the source in parallel this way the the time to pull the data will be less
j
Jhon Edison Bambague Calderon
12/14/2022, 7:41 AMHi @Eli Sigal, the timeout would be Airbyte? because at database level we have no timeout restrictions.
e
Eli Sigal
12/14/2022, 8:00 AMairbyte time out is a direct derivative of the source timeout i.e. it took to much time for the connector to run i.e. to much data on the source i.e. timeout
j
Jhon Edison Bambague Calderon
12/14/2022, 8:20 AMI understand but the error should be a timeout against the database and here the error is due to the worker according to the log file: "Sync worker failed" and "airbyte-source gobbler IOException: Connection timed out
Jhon Edison Bambague Calderon
12/14/2022, 8:23 AMThe Airbyte user timeout parameters on the source (DB) are not conditioned, they are all set to 0. Which means that the timeout is not configured on PostgreSQL
Jhon Edison Bambague Calderon
12/14/2022, 8:24 AMWhat do you recommend that we can review on the source side?
Jhon Edison Bambague Calderon
12/14/2022, 8:27 AMIts possible using this recommendation?
Jhon Edison Bambague Calderon
12/14/2022, 8:27 AM Copy code
# Max attempts per sync and max retries per attempt
MAX_SYNC_JOB_ATTEMPTS=3
# Time in days to reach a timeout to cancel the synchronization
MAX_SYNC_TIMEOUT_DAYS=3e
Eli Sigal
12/14/2022, 9:46 AMIt is because the workers also have a timeout if it takes to long.
try lowering the amount of data you pull from the source - this should fixed this
j
Jhon Edison Bambague Calderon
12/14/2022, 12:54 PMHi @Eli Sigal mmm we can't limit the amount of source data, is it possible to somehow increase those timeouts including the worker timeout?
e
Eli Sigal
12/14/2022, 3:43 PMHi.
I am not 100% sure since I did not try this before
Copy code
POST /v1/state/create_or_update2 Views