Hey! We are running on Kubernetes over EKS, deployed using he helm chart on version 0.39.41-alpha Every once in a while all our jobs start failing. We see very high memory usage (~4Gi RAM for 4 active jobs) in the worker pod, and after restarting the worker and server the jobs succeed. Unfortunately we lost the logs from the worker pod after we restarted it… Here are the logs from one of the failed sync jobs:

2022-10-18 11:34:34 ERROR i.a.w.g.DefaultCheckConnectionWorker(run):100 - Unexpected error while checking connection: 
io.airbyte.workers.exception.WorkerException: null
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:138) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.process.AirbyteIntegrationLauncher.check(AirbyteIntegrationLauncher.java:84) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:63) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.general.DefaultCheckConnectionWorker.run(DefaultCheckConnectionWorker.java:37) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.temporal.TemporalAttemptExecution.lambda$getWorkerThread$2(TemporalAttemptExecution.java:155) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at java.lang.Thread.run(Thread.java:1589) [?:?]
Caused by: java.lang.InterruptedException
	at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1681) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.pollFirst(LinkedBlockingDeque.java:515) ~[?:?]
	at java.util.concurrent.LinkedBlockingDeque.poll(LinkedBlockingDeque.java:677) ~[?:?]
	at io.airbyte.workers.process.KubePortManagerSingleton.take(KubePortManagerSingleton.java:67) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	at io.airbyte.workers.process.KubeProcessFactory.create(KubeProcessFactory.java:102) ~[io.airbyte-airbyte-workers-0.39.41-alpha.jar:?]
	... 5 more

@[DEPRECATED] Marcos Marx turned this message into Zendesk ticket 2905 to ensure timely resolution!

Hey there, could you try a newer version of Airbyte and let us know if it helps? We are at v0.40.16 today, might be some performance improvements

@Sunny Hashmi (Airbyte) That’s exactly what we’re doing now on staging.

🤞🏽 Let us know how it goes @Dudu Vaanunu

@Sunny Hashmi (Airbyte) We are struggling. The last version doesn’t seem stable as well. Unfortunately we might decide to migrate out of Airbyte if that’s the case. Is there a way to get more support?

Hey @Dudu Vaanunu sorry to hear about the bad experience 😞 I'm checking into that for you and will let you know. In the meantime, could you share the full sync log so I can help debug?

@Sunny Hashmi (Airbyte) Thanks for doing so. Is there a PR we can follow up with? Can we have a look on what was fixed? fyi @ns.

There have been a ton of performance improvements since 0.39.41-alpha 🙂 the next release is not cut yet so we can't see exactly what will be included, but you can see the history here: https://github.com/airbytehq/airbyte/releases/tag/v0.40.18

We are still experiencing this after upgrading to 0.40.17 We are running two worker replicas currently, and they tend to reach high memory usage and a large number of ports per pod. Would increasing worker replica count help alleviate this issue?? How else should we accomodate for scale? Thank you very much!

Hey @ns, if you are still experiencing this issue there are some tips in this doc: https://docs.airbyte.com/deploying-airbyte/on-kubernetes#increase-job-parallelism In a nutshell yes, you're correct, adding more worker replicas will help 🙂