Hello. I am trying to understand how airbyte deal ...
# replication-troubleshooting
r
Hello. I am trying to understand how airbyte deal with memory. Every time a conector reads a lot records, I need to increase the vm memory and cpu. I found the JOB_MAIN_CONTAINER_MEMORY_REQUEST and JOB_MAIN_CONTAINER_MEMORY_LIMIT. But Id like to understand how this works from source to dest. Example, if I set request and memory to 2gb but the first sync is 10gb of rows, how it will be handled? Rows goes to disk? And when airbyte goes to send this to dest, it will be batches of 2gb, or something like that? I really appreciate any link to docs, discussion or article that answers things like that. 🙂
Another doubt, if 2 syncs are running at same time and my limit is 2gb, then the jobs will use this 2gb each, or will be 1gb (2gb divided for 2 jobs)?