Why is airbyte-server using over 5 gigabytes of memory? This is deployed on kubernetes using helm. The airbyte stack has it's own node with 12-16GB of memory. Next step up is 32GB which adds significant cost when the data we are syncing is tiny: Sync Succeeded 7.12 MB | 12,751 records extracted | 12,751 records loaded

The high memory usage of the

airbyte-server

could be due to the Source worker within a Sync job. The Source worker reads up to 10,000 records in memory, which can present problems for database sources with tables that have large row sizes. For example, a table with an average row size of 0.5MBs will require 0.5 * 10000 / 1000 = 5GBs of RAM. Java connectors, which include all Source database connectors, follow Java's default behavior with container memory and will only use up to 1/4 of the host's allocated memory. On a Docker agent with 8GBs of RAM configured, a Java connector limits itself to 2Gbs of RAM and will see Out-of-Memory exceptions if this goes higher. The same applies to Kubernetes pods. You may want to customize this by setting

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to custom values. Please note that users currently need to over-specify memory resource for Java connectors due to this behavior. For more information, you can refer to the [Airbyte documentation on scaling](https://docs.airbyte.com/operator-guides/scaling-airbyte).

@kapa.ai Why is the memory not released when the sync job is completed?

Hmm, I don't know enough to give you a confident answer yet. The information extracts provided do not contain specific details about memory management after a sync job is completed in Airbyte. However, it's worth noting that memory management can depend on various factors including the specific source and destination connectors used, the configuration of your deployment, and the underlying system or platform Airbyte is running on. For more information on how Airbyte handles memory during sync jobs, you can refer to the [Airbyte Documentation on Scaling](https://docs.airbyte.com/operator-guides/scaling-airbyte). If you're experiencing specific issues with memory not being released, I would recommend reaching out to the Airbyte community on their [discussion forum](https://discuss.airbyte.io) or [Slack workspace](https://airbytehq.slack.com) for more targeted assistance.