Hi Everyone: I've installed an Airbyte instance on autopilot GKE cluster. Currently airbyte v2.0.1 (Helm chart v2.0.19). I've hit so many issues when syncing larger datasets, for example: • the OOM issues in job sync, • Failed to connect to airbyte-workload-api-server-svc.airbyte/xxx.xxx.xxx:8001 error • More frequent Google Bigquery insert errors ( 500 Internal Server Error, 400 Bad Request) Because of these, my syncs have been significantly slower and also less reliable than the legacy airbyte instance(v1.4.3) that was setup by abtctl( in a docker image on a small compute instance) Is it worthwhile to downgrade to a lower version of airbyte? Has anyone done it? Is it a big effort to do so?

With OOM, I would check your container logs and such and see both what it's provisioning and which of the containers is hitting the limit (i.e. destination or source). Then supply overrides via the database either at a connector or connection level (or bump your overall configured limits in the YAML, but this is usually overkill if you have workloads of different sizes). Don't forget that Autopilot mode will mutate the values (you'll see this in the logs if it's happening) to ensure that it has an autopilot-compatible ratio of memory-to-CPU. So sometimes the values you see actually being applied can be different than what's in your config.

Hi @Justin Beasley , Thank you for replying!!! With the OOM, I did this in my value.yaml

workloads:
      resources:
        mainContainer:
          memory:
            request: 2Gi
            limit: 32Gi
          cpu:
            request: 500m
            limit: 4000m
        replication:
          memory:
            request: 2Gi
            limit: 32Gi
          cpu:
            request: 500m
            limit: 4000m
        discover:
          memory:
            request: 2Gi
            limit: 8Gi
          cpu:
            request: 500m
            limit: 2000m   
  # Settings for Airbyte jobs pods
  jobs:
    kube: {}
    resources:
      requests:
        memory: 2Gi
        cpu: 1000m
      limits:
        cpu: 2000m
        memory: 8Gi

What I found was with v2.0.19, This doesnt overwrite my container resource limit configs... I tried redeploy the worker-launcher, but I still had

ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
 ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=1Gi, cpu=0.2}, additionalProperties={})
esources: ResourceRequirements(claims=[], limits={memory=2Gi, cpu=1}, requests={memory=2Gi, cpu=1}, additionalProperties={})

When I downgraded my airbyte cluster to v2.0.18 with the same value.yaml file, this works....

ResourceRequirements(claims=[], limits={memory=32Gi, cpu=4000m}, requests={memory=2Gi, cpu=500m}, additionalProperties={})

ResourceRequirements(claims=[], limits={memory=32Gi, cpu=4000m}, requests={memory=1Gi, cpu=500m}, additionalProperties={})

ResourceRequirements(claims=[], limits={memory=32Gi, cpu=4000m}, requests={memory=2Gi, cpu=500m}, additionalProperties={})

But then again my sync is still hitting an error

Failures: [ {
  "failureOrigin" : "source",
  "internalMessage" : "Source process exited with non-zero exit code 143",
  "externalMessage" : "Something went wrong within the source connector",
  "metadata" : {
    "attemptNumber" : 0,
    "jobId" : 50,
    "connector_command" : "read"
  },
  "stacktrace" : "io.airbyte.workers.internal.exception.SourceException: Source process exited with non-zero exit code 143\n\tat io.airbyte.container.orchestrator.worker.SourceReader.run(ReplicationTask.kt:210)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invokeSuspend(ReplicationWorker.kt:154)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.ReplicationWorker$runJobs$2$tasks$1$1.invoke(ReplicationWorker.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invokeSuspend(AsyncUtils.kt:22)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1$1.invoke(AsyncUtils.kt)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndspatched(Undispatched.kt:66)\n\tat kotlinx.coroutines.intrinsics.UndispatchedKt.startUndispatchedOrReturn(Undispatched.kt:43)\n\tat kotlinx.coroutines.BuildersKt__Builders_commonKt.withContext(Builders.common.kt:165)\n\tat kotlinx.coroutines.BuildersKt.withContext(Unknown Source)\n\tat io.airbyte.container.orchestrator.worker.util.AsyncUtils$runAsync$1.invokeSuspend(AsyncUtils.kt:21)\n\tat kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)\n\tat kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:100)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1583)\n\tSuppressed: io.airbyte.workers.internal.exception.DestinationException: Destination process exited with non-zero exit code 143\n\t\tat io.airbyte.container.orchestrator.worker.DestinationReader.run(ReplicationTask.kt:55)\n\t\t... 16 more\n",
  "timestamp" : 1773408345414

That's good to know that you at least got the resources being picked up (i went the downgrade to 1.x route). Some of the connector specific settings that are stored in the db (as mentioned above) can cause issues where those are picked up rather than the values.yaml config. However, this looks like a separate error specific to the source. Any more specific details in the logs? This is a pretty unhelpful trace (as usual) but...if it is still something with Replication, you can try adding

CONTAINER_ORCHESTRATOR_JAVA_OPTS

global:
  env_vars:
    CONTAINER_ORCHESTRATOR_JAVA_OPTS: "-XX:+ExitOnOutOfMemoryError -XX:MaxRAMPercentage=85.0"

Hi @Greg Corey Thanks for replying! Did you have any issue when downgrading to 1.x? This is my log before I saw the error above. Source - exit with code 143 Destination error then entire pod gets shut down

Great!