Hi Team, We have triggered 2 jobs where source is ...
# contributing-to-airbyte
n
Hi Team, We have triggered 2 jobs where source is GoogleAds(airbyte/source-google-ads:0.1.20) and destination is Snowflake(airbyte/destination-snowflake:0.3.14).All of a sudden airbyte server got crashed. We got an error saying "DEADLINE_EXCEEDED: deadline exceeded after 69.999905719s". We even saw the related issues on github and we are running on the suggested configuration. It used to work fine in Airbyte version: 0.29.22. But we are facing this issue after we have upgraded to 0.35.15. Airbyte Version: 0.35.15-alpha Server Details:     Ubuntu 20.04.3 LTS     RAM: 16 GB     4 Core CPU     DiskSpace 200gb Im adding the job logs for better understanding:
Copy code
2022-02-07 09:24:27 WARN i.t.i.r.GrpcSyncRetryer(retry):56 - Retrying
after failure
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded
after 69.999905719s. [closed=[],
open=[[remote_addr=airbyte-temporal/192.168.144.6:7233]]]
at
io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262)
~[grpc-stub-1.42.1.jar:1.42.1]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243)
~[grpc-stub-1.42.1.jar:1.42.1]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156)
~[grpc-stub-1.42.1.jar:1.42.1]
at
io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2642)
~[temporal-servicecl
ient-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.lambda$getInstanceCloseEvent$0(WorkflowClientLongPollHelper.java:143)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.retryer.GrpcSyncRetryer.retry(GrpcSyncRetryer.java:61)
~[temporal-serviceclient-1.6.0.jar:?]
at
io.temporal.internal.retryer.GrpcRetryer.retryWithResult(GrpcRetryer.java:51)
~[temporal-serviceclient-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.getInstanceCloseEvent(WorkflowClientLongPollHelper.java:131)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.client.WorkflowClientLongPollHelper.getWorkflowExecutionResult(WorkflowClientLongPollHelper.java:72)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.client.RootWorkflowClientInvoker.getResult(RootWorkflowClientInvoker.java:93)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:243)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:225)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270)
~[temporal-sdk-1.6.0.jar:?]
at
io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178)
~[temporal-sdk-1.6.0.jar:?]
at jdk.proxy2.$Proxy40.run(Unknown Source) ~[?:?]
at
io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:148)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:439)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:147)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.workers.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:83)
~[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at io.airbyte.workers.worker_run.WorkerRun.call(WorkerRun.java:51)
[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at io.airbyte.workers.worker_run.WorkerRun.call(WorkerRun.java:22)
[io.airbyte-airbyte-workers-0.35.15-alpha.jar:?]
at
io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:94)
[io.airbyte-airbyte-commons-0.35.15-alpha.jar:?]
at
io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:78)
[io.airbyte-airbyte-commons-0.35.15-alpha.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
[?:?]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
[?:?]
at java.lang.Thread.run(Thread.java:833) [?:?]
w
Hi, I had the same issue with the same source and destination - however running airbyte version 30.15 - the server crashed it exceeded cpu resources - I since switched from AWS t3 medium to t3xlarge and since then utilisation is more reasonable
Copy code
2022-02-01 15:48:22 INFO () DefaultReplicationWorker(lambda$getReplicationRunnable$2):203 - Records read: 10000
2022-02-01 15:48:23 INFO () DefaultAirbyteStreamFactory(lambda$create$0):53 - 2022-02-01 15:48:23 [32mINFO[m i.a.i.d.s.SnowflakeSqlOperations(insertRecordsInternal):57 - {} - actual size of batch: 9819
2022-02-01 15:59:03 WARN () ActivityExecutionContextImpl(doHeartBeat):153 - Heartbeat failed
io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: deadline exceeded after 9.987127496s. [closed=[], open=[[remote_addr=airbyte-temporal/172.25.0.8:7233]]]
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.recordActivityTaskHeartbeat(WorkflowServiceGrpc.java:2710) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.sendHeartbeatRequest(ActivityExecutionContextImpl.java:203) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.doHeartBeat(ActivityExecutionContextImpl.java:147) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.ActivityExecutionContextImpl.heartbeat(ActivityExecutionContextImpl.java:108) ~[temporal-sdk-1.0.4.jar:?]
	at io.airbyte.workers.temporal.CancellationHandler$TemporalCancellationHandler.checkAndHandleCancellation(CancellationHandler.java:48) ~[io.airbyte-airbyte-workers-0.30.18-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getCancellationChecker$3(TemporalAttemptExecution.java:192) ~[io.airbyte-airbyte-workers-0.30.18-alpha.jar:?]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
	at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]