Channels
d
Davin Chia (Airbyte)
09/20/2021, 7:24 AM
@gunu we are going to do some stress testing this week and I want to include the CDC failure case you are seeing and hopefully get more data to fix this. I was hoping for more info on the table's schema.
(e.g. one connection, one table, 120m rows, 50gb data, mysql -> snowflake)This is what I have now. How many columns are there and what's the average size of each row?
See attached CSV with some schema specs
• attempted all of these in one connection, but it ends up hanging after reading everything
• removed TABLE A and B from the connection and managed to achieve successful CDC syncs
• attempted TABLE A in seperate connection (same CDC MySQL source) but that job just hangs
u
user
09/20/2021, 7:44 AMand note: CDC failure for me means job hangs (source container persists as ~100% CPU) and worker doesnt actually its finished state or doesnt trigger the next phase
u
user
09/20/2021, 7:49 AMGreat, thank you!
u
user
09/20/2021, 7:56 AMso table A is 5GB and not 50GB?
u
user
09/20/2021, 8:02 AMoh wow 🤦 yes correct! i’ll need to update my comments elsewhere. the “biggest” table i’m having problems with is only 5gb
u
user
09/20/2021, 8:04 AMwe've definitely seen syncs with larger tables so this is very interesting
d
Davin Chia (Airbyte)
09/20/2021, 8:05 AM
was just thinking the same thing
u
user
09/20/2021, 8:41 PMI successfully ran a connector with all tables excluding table A and table B.
now i just ran a seperate connector wiht table B (1gb and 60m rows) and it didn’t work. provided logs to show how it hangs at the end
u
user
09/22/2021, 11:08 AMthe actual schema of the 60 million row (1gb) table that fails/hangs on its own connection. mysql (version 5.7.33) CDC to snowflake.
u
user
09/22/2021, 1:15 PMand just confirming that i ran this same table as standard replication (not CDC) and it works fine.
curious though as the first sync is full refresh and not really using CDC anyways no?
m
Mané Rom
09/22/2021, 1:47 PMand lastly. the seemingly 1GB of source data (with no json data, only int types - see schema CSV above) appears to conflate to 17GB in the airbyte job success status. (in snowflake the query seemed to scan 5GB of data)
u
user
09/22/2021, 1:57 PMside note. could this issue be affecting these connectors that use AWS S3 Staging
u
user
09/23/2021, 12:33 AMand just confirming that i ran this same table as standard replication (not CDC) and it works fine.
curious though as the first sync is full refresh and not really using CDC anyways no?althought it’s a full refresh ‘type’, it uses a different mechanism under the hood since it’s reading from DB WAL logs and not sql queries
and lastly. the seemingly 1GB of source data (with no json data, only int types - see schema CSV above) appears to conflate to 17GB in the airbyte job success status. (in snowflake the query seemed to scan 5GB of data)interesting.. good info!
u
user
09/26/2021, 3:10 AMUpdates: I retried this over the weekend and it appears to get over the hanging stage now 🎉 buuuut running into this error at the normalization phase
Copy code
2021-09-26 02:45:03 WARN () GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.0.4.jar:?]
at com.sun.proxy.$Proxy38.run(Unknown Source) ~[?:?]
at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:124) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:144) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:123) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) ~[io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]3 Views