Davin Chia (Airbyte)
(e.g. one connection, one table, 120m rows, 50gb data, mysql -> snowflake)This is what I have now. How many columns are there and what's the average size of each row?
user
09/20/2021, 7:44 AMuser
09/20/2021, 7:49 AMuser
09/20/2021, 7:56 AMuser
09/20/2021, 8:02 AMuser
09/20/2021, 8:04 AMDavin Chia (Airbyte)
user
09/20/2021, 8:41 PMuser
09/22/2021, 11:08 AMuser
09/22/2021, 1:15 PMMané Rom
09/22/2021, 1:47 PMuser
09/22/2021, 1:57 PMuser
09/23/2021, 12:33 AMand just confirming that i ran this same table as standard replication (not CDC) and it works fine.
curious though as the first sync is full refresh and not really using CDC anyways no?althought it’s a full refresh ‘type’, it uses a different mechanism under the hood since it’s reading from DB WAL logs and not sql queries
and lastly. the seemingly 1GB of source data (with no json data, only int types - see schema CSV above) appears to conflate to 17GB in the airbyte job success status. (in snowflake the query seemed to scan 5GB of data)interesting.. good info!
user
09/26/2021, 3:10 AM2021-09-26 02:45:03 WARN () GrpcRetryer(retryWithResult):152 - Retrying after failure
io.grpc.StatusRuntimeException: UNAVAILABLE: Network closed for unknown reason
at io.grpc.stub.ClientCalls.toStatusRuntimeException(ClientCalls.java:262) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.grpc.stub.ClientCalls.getUnchecked(ClientCalls.java:243) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.grpc.stub.ClientCalls.blockingUnaryCall(ClientCalls.java:156) ~[grpc-stub-1.40.0.jar:1.40.0]
at io.temporal.api.workflowservice.v1.WorkflowServiceGrpc$WorkflowServiceBlockingStub.getWorkflowExecutionHistory(WorkflowServiceGrpc.java:2638) ~[temporal-serviceclient-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.lambda$getInstanceCloseEvent$1(WorkflowExecutionUtils.java:256) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.GrpcRetryer.retryWithResult(GrpcRetryer.java:127) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.getInstanceCloseEvent(WorkflowExecutionUtils.java:244) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.common.WorkflowExecutionUtils.getWorkflowExecutionResult(WorkflowExecutionUtils.java:132) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:346) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowStubImpl.getResult(WorkflowStubImpl.java:328) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.startWorkflow(WorkflowInvocationHandler.java:315) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler$SyncWorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:270) ~[temporal-sdk-1.0.4.jar:?]
at io.temporal.internal.sync.WorkflowInvocationHandler.invoke(WorkflowInvocationHandler.java:178) ~[temporal-sdk-1.0.4.jar:?]
at com.sun.proxy.$Proxy38.run(Unknown Source) ~[?:?]
at io.airbyte.workers.temporal.TemporalClient.lambda$submitSync$3(TemporalClient.java:124) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalClient.execute(TemporalClient.java:144) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.workers.temporal.TemporalClient.submitSync(TemporalClient.java:123) ~[io.airbyte-airbyte-workers-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.TemporalWorkerRunFactory.lambda$createSupplier$0(TemporalWorkerRunFactory.java:67) ~[io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:65) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.scheduler.app.worker_run.WorkerRun.call(WorkerRun.java:43) [io.airbyte.airbyte-scheduler-app-0.29.22-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.execute(LifecycledCallable.java:114) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
at io.airbyte.commons.concurrency.LifecycledCallable.call(LifecycledCallable.java:98) [io.airbyte-airbyte-commons-0.29.22-alpha.jar:?]
at java.util.concurrent.FutureTask.run(FutureTask.java:264) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
at java.lang.Thread.run(Thread.java:832) [?:?]