how to limit sync using too much ram ?
# ask-community-for-troubleshooting
y
how to limit sync using too much ram ?
k
A message from kapa.ai
To limit the RAM usage for sync jobs in Airbyte, you can set the
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. These variables define the minimum and maximum RAM usage for the job container. For example, if you're using Docker, you can set these environment variables in your
.env
file or directly in the
docker-compose.yml
file. If you're using Kubernetes, you can set them in your deployment configuration. Keep in mind that limiting the memory usage too much may lead to Out-of-Memory (OOM) exceptions. It's important to find a balance between the available resources and the requirements of your sync jobs. Here's an extract from the documentation that explains this in more detail: > Our Java connectors currently follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. e.g. On a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2Gbs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. You may want to customize this by setting
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. [Source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)
y
In my case it is using all 4 gb remaining left @kapa.ai
k
A message from kapa.ai
Sorry @Yusuf Mirkar, I am experiencing especially high traffic right now, please try again. 🧑‍💻
y
@kapa.ai In my case it is using all 4 gb remaining left
k
A message from kapa.ai
A message from kapa.ai
Hmm, I don't know enough to give you a confident answer yet. However, if you're looking to configure resource limits for Airbyte, you can refer to this [previous support conversation](https://airbytehq.slack.com/archives/C021JANJ6TY/p1674163243149269) which explains how to set up resource limits in the
.env
file when deploying with Docker.
To limit the memory usage of Airbyte, you can set resource limits in the Docker Compose file if you're deploying with Docker. You can find more information on how to do this in the [Docker documentation](https://docs.docker.com/compose/compose-file/compose-file-v3/#resources). If you're deploying Airbyte on Kubernetes, you can set resource limits for core container pods and connector pods. For core container pods, modify the
kube/overlays/stable-with-resource-limits/set-resource-limits.yaml
file and apply the changes using
kubectl apply -k kube/overlays/stable-with-resource-limits
. For connector pods, configure the
Docker resource limits
section of the
.env
file in the
kube/overlays
directory. More details can be found in the [Airbyte documentation](https://docs.airbyte.com/deploying-airbyte/on-kubernetes#set-resource-limits). Remember to adjust the memory limits according to your requirements and available resources.