https://linen.dev logo
d

Davin Chia (Airbyte)

09/20/2021, 7:24 AM
@gunu we are going to do some stress testing this week and I want to include the CDC failure case you are seeing and hopefully get more data to fix this. I was hoping for more info on the table's schema.
(e.g. one connection, one table, 120m rows, 50gb data, mysql -> snowflake)
This is what I have now. How many columns are there and what's the average size of each row?
See attached CSV with some schema specs • attempted all of these in one connection, but it ends up hanging after reading everything • removed TABLE A and B from the connection and managed to achieve successful CDC syncs • attempted TABLE A in seperate connection (same CDC MySQL source) but that job just hangs
u

user

09/20/2021, 7:44 AM
and note: CDC failure for me means job hangs (source container persists as ~100% CPU) and worker doesnt actually its finished state or doesnt trigger the next phase
u

user

09/20/2021, 7:49 AM
Great, thank you!
u

user

09/20/2021, 7:56 AM
so table A is 5GB and not 50GB?
u

user

09/20/2021, 8:02 AM
oh wow 🤦 yes correct! i’ll need to update my comments elsewhere. the “biggest” table i’m having problems with is only 5gb
u

user

09/20/2021, 8:04 AM
we've definitely seen syncs with larger tables so this is very interesting
d

Davin Chia (Airbyte)

09/20/2021, 8:05 AM
was just thinking the same thing
u

user

09/20/2021, 8:41 PM
I successfully ran a connector with all tables excluding table A and table B. now i just ran a seperate connector wiht table B (1gb and 60m rows) and it didn’t work. provided logs to show how it hangs at the end
u

user

09/22/2021, 11:08 AM
the actual schema of the 60 million row (1gb) table that fails/hangs on its own connection. mysql (version 5.7.33) CDC to snowflake.
u

user

09/22/2021, 1:15 PM
and just confirming that i ran this same table as standard replication (not CDC) and it works fine. curious though as the first sync is full refresh and not really using CDC anyways no?
m

Mané Rom

09/22/2021, 1:47 PM
and lastly. the seemingly 1GB of source data (with no json data, only int types - see schema CSV above) appears to conflate to 17GB in the airbyte job success status. (in snowflake the query seemed to scan 5GB of data)
u

user

09/22/2021, 1:57 PM
side note. could this issue be affecting these connectors that use AWS S3 Staging
u

user

09/23/2021, 12:33 AM
and just confirming that i ran this same table as standard replication (not CDC) and it works fine.
curious though as the first sync is full refresh and not really using CDC anyways no?
althought it’s a full refresh ‘type’, it uses a different mechanism under the hood since it’s reading from DB WAL logs and not sql queries
and lastly. the seemingly 1GB of source data (with no json data, only int types - see schema CSV above) appears to conflate to 17GB  in the airbyte job success status. (in snowflake the query seemed to scan 5GB of data)
interesting.. good info!
u

user

09/26/2021, 3:10 AM
Updates: I retried this over the weekend and it appears to get over the hanging stage now 🎉 buuuut running into this error at the normalization phase
Copy code
2021-09-26 02:45:03 WARN () GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
	at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.40.0.jar:1.40.0]
	at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) ~[temporal-sdk-1.0.4.jar:?]
	at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.0.4.jar:?]
	at com.sun.proxy.$Proxy38.run(Unknown Source) ~[?:?]
	at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:124) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:144) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:123) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) ~[io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
	at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
	at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
	at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
	at java.lang.Thread.run(Thread.java:832) [?:?]
3 Views