https://linen.dev logo
#troubleshooting
Title
# troubleshooting
n

Nahid Oulmi

03/14/2022, 10:32 AM
Is this your first time deploying Airbyte: No OS Version / Instance: Debian GNU/Linux, 10 (buster), amd64 built on, GCP ,e2-standard-4 Memory / Disk: 4 CPU, 8Gb RAM Deployment: Docker Airbyte Version: 0.35.7-alpha Step: Setting up resources Description: My Airbyte jobs take too much of my VM’s RAM ; at some point the VM is down, with no access to the Airbyte’s UI nor via SSH; the only fix available is to shutdown & restart the instance. In order to avoid that, I would like to set a global RAM consumption threshold for all Airbyte, but I am not sure which solution is better ; • The Docker
-memory
parameter seems a good option but I am not sure if it works well within Airbyte deployments : https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory • The Airbyte specific parameter
JOB_MAIN_CONTAINER_MEMORY_LIMIT
works at a job level if I am not mistaken ; As I don’t know how many Airbyte jobs can be triggered at the same time, if I have 10 jobs consuming only 1GB of RAM at the same time it will cause the same issue, which is why I would prefer to set a global RAM threshold. What do you think would be the best option ?
2 Views