Hey! I’m syncing from SurveyMonkey to BigQuery. It...
# replication-troubleshooting
d
Hey! I’m syncing from SurveyMonkey to BigQuery. It seems to sync completely (finishes normalization), but the whole sync fails with an error (larger stack trace in 🧵):
Copy code
ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
Additional info: OSS v0.40.17, running on an Autopilot GKE Cluster _SurveyMonkey source_: v0.1.14 _BigQuery destination_: v1.2.13 Takes about 1.5 hours, 132k records/211MB
✅ 2
Copy code
2023-02-01 08:03:59 ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:396) ~[?:?]
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2073) ~[?:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:189) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.17.jar:?]
	at java.lang.Thread.run(Thread.java:1589) ~[?:?]
	Suppressed: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.
		at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:151) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
		at io.airbyte.workers.general.DefaultReplicationWorker.run(DefaultReplicationWorker.java:71) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
		at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$4(TemporalAttemptExecution.java:190) ~[io.airbyte-airbyte-workers-0.40.17.jar:?]
		at java.lang.Thread.run(Thread.java:1589) ~[?:?]
Caused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:376) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	... 1 more
Caused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.
	at io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
	at io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:374) ~[io.airbyte-airbyte-commons-worker-0.40.17.jar:?]
	at java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
	... 1 more
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):297 - Source did not output any state messages
2023-02-01 08:03:59 WARN i.a.w.g.DefaultReplicationWorker(run):305 - State capture: No new state, falling back on input state: io.airbyte.config.State@4e7b7f66[state={}]
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):316 - sync summary: {
  "status" : "failed",
  "recordsSynced" : 132570,
  "bytesSynced" : 211718562,
  "startTime" : 1675233024153,
  "endTime" : 1675238639242,
  "totalStats" : {
    "recordsEmitted" : 132570,
    "bytesEmitted" : 211718562,
    "sourceStateMessagesEmitted" : 0,
    "destinationStateMessagesEmitted" : 0,
    "recordsCommitted" : 0,
    "meanSecondsBetweenStateMessageEmittedandCommitted" : 0,
    "replicationStartTime" : 1675233024153,
    "replicationEndTime" : 1675238639232,
    "sourceReadStartTime" : 1675233077977,
    "sourceReadEndTime" : 0,
    "destinationWriteStartTime" : 1675233132186,
    "destinationWriteEndTime" : 1675238639152
  },
  "streamStats" : [ {
    "streamName" : "survey_pages",
    "stats" : {
      "recordsEmitted" : 246,
      "bytesEmitted" : 59022
    }
  }, {
    "streamName" : "survey_questions",
    "stats" : {
      "recordsEmitted" : 646,
      "bytesEmitted" : 614325
    }
  }, {
    "streamName" : "survey_responses",
    "stats" : {
      "recordsEmitted" : 131678,
      "bytesEmitted" : 211045215
    }
  } ]
}
2023-02-01 08:03:59 INFO i.a.w.g.DefaultReplicationWorker(run):317 - failures: [ {
  "failureOrigin" : "source",
  "internalMessage" : "io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!",
  "externalMessage" : "Something went wrong within the source connector",
  "metadata" : {
    "attemptNumber" : 1,
    "jobId" : 1379,
    "connector_command" : "read"
  },
  "stacktrace" : "java.util.concurrent.CompletionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat java.base/java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:315)\n\tat java.base/java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:320)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1807)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1589)\nCaused by: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped!\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:376)\n\tat java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1804)\n\t... 3 more\nCaused by: io.airbyte.workers.exception.WorkerException: Source process exit with code 137. This warning is normal if the job was cancelled.\n\tat io.airbyte.workers.internal.DefaultAirbyteSource.close(DefaultAirbyteSource.java:135)\n\tat io.airbyte.workers.general.DefaultReplicationWorker.lambda$getReplicationRunnable$6(DefaultReplicationWorker.java:374)\n\t... 4 more\n",
  "timestamp" : 1675238626096
} ]
Hmm the read pod seems to terminate with an OOM error:
Copy code
State:          Terminated
      Reason:       OOMKilled
      Exit Code:    137
m
Hello @Dominik Mall can you share the complete log file here? Also meanwhile you can try fewer streams, a more recent start date or increase the resource for the POD to check if solve the OOM
d
Hi @Marcos Marx (Airbyte)! Here are the full logs, it’s pretty clear that
survey_responses
is the problematic stream (which also is important) i tried a more recent start date and it was successful (but then is missing data we need). I’m not sure how to increase the resource request/limit for that pod, looking into it now
u
d
Thanks - managed to get there after battling with unhelpful errors - in the end had to tear down the whole deployment in order to get those env vars to apply properly! Sync is currently running with increased memory request.. lets see how it goes 🤞
I can confirm that increasing the memory request solved the issue 🎉