Yusuf Mirkar
05/25/2023, 9:50 AMuser
05/25/2023, 9:50 AMuser
05/25/2023, 9:50 AMJOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables. Source: [Scaling Airbyte - Memory](https://docs.airbyte.com/operator-guides/scaling-airbyte)Yusuf Mirkar
05/25/2023, 7:00 PMuser
05/25/2023, 7:00 PMuser
05/25/2023, 7:01 PMJOB_MAIN_CONTAINER_MEMORY_REQUEST
environment variable sets the requested memory for the main container, but it doesn't guarantee that the container will use the exact amount specified. The actual memory usage depends on the connector and the data being processed. In the case of Java connectors, they follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory by default. For example, on a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2GBs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods [source](https://docs.airbyte.com/operator-guides/scaling-airbyte). If you want to customize the memory usage, you can set both JOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. However, keep in mind that the actual memory usage may still vary depending on the connector and the data being processed.