https://linen.dev logo
Title
y

Yusuf Mirkar

05/25/2023, 9:50 AM
what does this mean ? Note that all Source database connectors are Java connectors. This means that users currently need to over-specify memory resource for Java connectors.
u

user

05/25/2023, 9:50 AM
A message from kapa.ai
u

user

05/25/2023, 9:50 AM
This statement means that all Source database connectors in Airbyte are implemented using Java. Due to the default behavior of Java connectors, they use up to 1/4 of the host's allocated memory. This can lead to situations where users need to allocate more memory than actually required for the Java connectors to avoid running out of memory. For example, if a Docker agent has 8GB of RAM configured, a Java connector will limit itself to 2GB of RAM and may encounter Out-of-Memory exceptions if the memory usage goes higher. To avoid this, users may need to allocate more memory than necessary for the Java connectors by customizing the
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables. Source: [Scaling Airbyte - Memory](https://docs.airbyte.com/operator-guides/scaling-airbyte)
y

Yusuf Mirkar

05/25/2023, 7:00 PM
@kapa.ai I had allocated 3 gb in JOB_MAIN_CONTAINER_MEMORY_REQUEST, but it was using 2.3 gb which is not 1/4th of allocated gb
u

user

05/25/2023, 7:00 PM
A message from kapa.ai
u

user

05/25/2023, 7:01 PM
It's important to note that the
JOB_MAIN_CONTAINER_MEMORY_REQUEST
environment variable sets the requested memory for the main container, but it doesn't guarantee that the container will use the exact amount specified. The actual memory usage depends on the connector and the data being processed. In the case of Java connectors, they follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory by default. For example, on a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2GBs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods [source](https://docs.airbyte.com/operator-guides/scaling-airbyte). If you want to customize the memory usage, you can set both
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values. However, keep in mind that the actual memory usage may still vary depending on the connector and the data being processed.