Nahid Oulmi
03/14/2022, 10:32 AM-memory
parameter seems a good option but I am not sure if it works well within Airbyte deployments : https://docs.docker.com/config/containers/resource_constraints/#limit-a-containers-access-to-memory
• The Airbyte specific parameter JOB_MAIN_CONTAINER_MEMORY_LIMIT
works at a job level if I am not mistaken ; As I don’t know how many Airbyte jobs can be triggered at the same time, if I have 10 jobs consuming only 1GB of RAM at the same time it will cause the same issue, which is why I would prefer to set a global RAM threshold.
What do you think would be the best option ?