Yusuf Mirkar
05/18/2023, 3:06 PMkapa.ai
05/18/2023, 3:06 PMkapa.ai
05/18/2023, 3:07 PMJOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. These variables define the minimum and maximum RAM usage for the job container. For example, if you're using Docker, you can set these environment variables in your .env
file or directly in the docker-compose.yml
file. If you're using Kubernetes, you can set them in your deployment configuration. Keep in mind that limiting the memory usage too much may lead to Out-of-Memory (OOM) exceptions. It's important to find a balance between the available resources and the requirements of your sync jobs. Here's an extract from the documentation that explains this in more detail: > Our Java connectors currently follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. e.g. On a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2Gbs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. You may want to customize this by setting JOB_MAIN_CONTAINER_MEMORY_REQUEST
and JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. [Source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)Yusuf Mirkar
05/18/2023, 3:09 PMkapa.ai
05/18/2023, 3:10 PMkapa.ai
05/18/2023, 3:15 PMYusuf Mirkar
05/18/2023, 5:27 PMkapa.ai
05/18/2023, 5:27 PMkapa.ai
05/18/2023, 5:27 PMkapa.ai
05/18/2023, 5:28 PM.env
file when deploying with Docker.kapa.ai
05/18/2023, 5:28 PMkube/overlays/stable-with-resource-limits/set-resource-limits.yaml
file and apply the changes using kubectl apply -k kube/overlays/stable-with-resource-limits
. For connector pods, configure the Docker resource limits
section of the .env
file in the kube/overlays
directory. More details can be found in the [Airbyte documentation](https://docs.airbyte.com/deploying-airbyte/on-kubernetes#set-resource-limits). Remember to adjust the memory limits according to your requirements and available resources.