JOB_MAIN_CONTAINER_MEMORY_REQUEST=1g JOB_MAIN_CONTAINER_MEMORY_LIMIT=1g but source container keeps consuming more than 1gb

The

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables are used to set the memory request and limit for the main job container in Airbyte. However, it's important to note that the main memory consumer within a Sync job is the Source worker, which reads up to 10,000 records in memory. This can present problems for database sources with tables that have large row sizes. For example, a table with an average row size of 0.5MBs will require 0.5 * 10000 / 1000 = 5GBs of RAM. If your source data has large row sizes, it might exceed the memory limit you've set. Java connectors in Airbyte follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. If this limit is exceeded, you might see Out-of-Memory exceptions. If you're experiencing issues with memory consumption, you might need to adjust these environment variables to allocate more memory to your job containers. However, be aware that all Source database connectors are Java connectors, which means you might need to over-specify memory resources for Java connectors. For more information, you can refer to the [Airbyte documentation on scaling](https://docs.airbyte.com/operator-guides/scaling-airbyte) and [configuring connector resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources).

@kapa.ai if worker container is assigned 3gb of ram what will be the source and destination container limit?

The memory limit for the Source and Destination containers in Airbyte is determined by the

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variable. If you set this variable to 3GB, both the Source and Destination containers will have a maximum RAM usage of 3GB. However, it's important to note that Java connectors, which include all Source database connectors, follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. For example, if a Docker agent has 8GBs of RAM configured, a Java connector limits itself to 2GBs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. If you want to customize this, you can set

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to custom values. Please refer to the [Scaling Airbyte](https://docs.airbyte.com/operator-guides/scaling-airbyte/) and [Configuring Connector Resources](https://docs.airbyte.com/operator-guides/configuring-connector-resources/) guides for more information.