Hi team! We deployed Airbyte (v0.40.17) on kubernetes using Helm, and make some test with S3 --> Snowflake and PG --> Snowflake and its worked. When we try to transfer a big table From PG to SF using CDC, PG (v1.0.34), the first batch of 324.43 GB, and 54,933,278 rows flowed well. But when the connector run again, the workers failed:

ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException:io.airbyte.workers.general.DefaultReplicationWorker$DestinationException: Destination process message delivery failed.

And this other log:

2022-12-24 01:05:55 ERROR i.a.w.g.DefaultReplicationWorker(run):196 - Sync worker failed.
java.util.concurrent.ExecutionException: io.airbyte.workers.general.DefaultReplicationWorker$SourceException: Source cannot be stopped

We running this over a m6i.xlarge. So, first, someone have any idea of whats wrong? second, I would like to know how Airbyte works behind when make the CDC or the behavior of Airbyte in this process. How use the workers and compare the data what already are in the destination and the new data. Best, Ignacio.

Hello, are you still experiencing this issue? Can you try the latest version as there have been some improvements, and you still get the error please share the full sync log?

hey, if you are having this issue, the solution for me was to increase resources for the sync workers. You can do that by changing the limits of the container inside Airbyte DB, this doc helped me to do that: https://docs.airbyte.com/operator-guides/configuring-connector-resources/#configuring-connector-specific-requirements

We’re running into this problem now. We’ve been using Airbyte for a few months now successfully, but are now trying to do an initial sync of a new table that’s bigger than any we currently have and the sync is erroring out. I’m assuming incremental updates won’t be an issue but we need to get this working first.

hey @Matt Webster! After you are connect to airbyte database, you run this command to find the connector definition id of the connector you are having problems:

airbyte=# select docker_repository,resource_requirements from actor_definition where actor_definition.docker_repository like '%github';

Thanks Gabriel! I’m thinking that memory is the limiting factor for us. What did you end up setting your job-specific memory limit to? Did you double the default? Triple? Hoping for any guidance you can offer…

I'm trying to run SHOPIFY with a historical start date and it keeps erroring out - it drains all the memory on my EC2 instance. I'm going to relaunch airbyte with these limits set - I'm curious why they default to NONE in the

.env

file provided in the repo. I also don't understand why setting the limit will help. If airbyte needs 20GB of memory, and I"m on an instance with 8GB Ram and 100GB storage, why does it not borrow from the storage space? Further, if I set limits as outlined here, what does it do when a job hits it's limit. Let's say I set a 1GB memory limit, and my job needs 20GB. Does the job fail?