Dominik Mall
02/01/2023, 8:33 AMERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
Additional info:
OSS v0.40.17, running on an Autopilot GKE Cluster
_SurveyMonkey source_: v0.1.14
_BigQuery destination_: v1.2.13
Takes about 1.5 hours, 132k records/211MBDominik Mall
02/01/2023, 8:35 AM2023-02-01 08:03:59 ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:189) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.17.jar:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.
at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:151) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.17.jar:?]
at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:376) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
... 1 more
Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.
at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:374) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
... 1 more
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):297 - Source did not output any state messages
2023-02-01 08:03:59 WARN i.a.w.g.DefaultReplicationWorker(run):305 - State capture: No new state, falling back on input state: io.airbyte.config.State@4e7b7f66[state={}]
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):316 - sync summary: {
"status" : "failed",
"recordsSynced" : 132570,
"bytesSynced" : 211718562,
"startTime" : 1675233024153,
"endTime" : 1675238639242,
"totalStats" : {
"recordsEmitted" : 132570,
"bytesEmitted" : 211718562,
"sourceStateMessagesEmitted" : 0,
"destinationStateMessagesEmitted" : 0,
"recordsCommitted" : 0,
"meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
"replicationStartTime" : 1675233024153,
"replicationEndTime" : 1675238639232,
"sourceReadStartTime" : 1675233077977,
"sourceReadEndTime" : 0,
"destinationWriteStartTime" : 1675233132186,
"destinationWriteEndTime" : 1675238639152
},
"streamStats" : [ {
"streamName" : "survey_pages",
"stats" : {
"recordsEmitted" : 246,
"bytesEmitted" : 59022
}
}, {
"streamName" : "survey_questions",
"stats" : {
"recordsEmitted" : 646,
"bytesEmitted" : 614325
}
}, {
"streamName" : "survey_responses",
"stats" : {
"recordsEmitted" : 131678,
"bytesEmitted" : 211045215
}
} ]
}
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):317 - failures: [ {
"failureOrigin" : "source",
"internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!",
"externalMessage" : "Something went wrong within the source connector",
"metadata" : {
"attemptNumber" : 1,
"jobId" : 1379,
"connector_command" : "read"
},
"stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:376)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:374)\n\t... 4 more\n",
"timestamp" : 1675238626096
} ]Dominik Mall
02/01/2023, 8:45 AMState: Terminated
Reason: OOMKilled
Exit Code: 137Marcos Marx (Airbyte)
02/01/2023, 8:46 PMDominik Mall
02/02/2023, 3:10 AMsurvey_responses is the problematic stream (which also is important)
i tried a more recent start date and it was successful (but then is missing data we need).
I’m not sure how to increase the resource request/limit for that pod, looking into it nowuser
02/02/2023, 8:30 AMDominik Mall
02/02/2023, 8:32 AMDominik Mall
02/02/2023, 2:19 PM