I just upgraded to an instance with 32gb of RAM (from 8gb), but looking at the containers Airbyte spawns, it's still allocating only 2gb of ram for the read/write containers. @kapa.ai

Airbyte's Java connectors, which include all Source database connectors, follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. For example, on a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2GBs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. To customize this, you can set the

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to custom values. These variables are applicable to both Docker and Kubernetes deployments. Here's an example of how a user set them in a previous support conversation:

bash JOB_MAIN_CONTAINER_MEMORY_REQUEST=28g JOB_MAIN_CONTAINER_MEMORY_LIMIT=28g

This user reported an increase in sync speed after increasing these variables and the instance size. Please note that all Source database connectors are Java connectors. This means that users currently need to over-specify memory resource for Java connectors. You can confirm the right configuration is being detected by navigating to the Job in the UI, and looking for the log. On Docker, the log will look something like this:

bash Creating docker container = destination-e2e-test-write-39-0-vnqtl with resources io.airbyte.config.ResourceRequirements@1d86d7c9[cpuRequest=<null>,cpuLimit=<null>,memoryRequest=200Mi,memoryLimit=200Mi]

On Kubernetes, the log will look something like this:

bash 2022-08-12 01:22:20 INFO i.a.w.p.KubeProcessFactory(create):100 - Attempting to start pod = source-intercom-check-480195-0-abvnr for airbyte/source-intercom:0.1.24 with resources io.airbyte.config.ResourceRequirements@11cc9fb9[cpuRequest=2,cpuLimit=2,memoryRequest=200Mi,memoryLimit=200Mi]

For more information, you can refer to the [Airbyte documentation on configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources) and [scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte). Sources: - [Airbyte Documentation: Configuring Connector Resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources) - [Airbyte Documentation: Scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte) - [Previous Support Conversation in Slack](https://airbytehq.slack.com/archives/C021JANJ6TY/p1675218730001299)

If I change the environment variable, do I just have to restart airbyte?

A message from kapa.ai

I'm looking at the logs and its looking like its still using the old values after changing the env variables and restarting