airbyte-worker is taking too much memory, it just increases to 8 gb until it uses all

A message from kapa.ai

It seems you're experiencing an issue with increasing memory usage in the Airbyte worker. This could be related to the Source worker, which reads up to 10,000 records in memory and can cause high memory usage for large row sizes in database sources ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)). One possible workaround is to restart the airbyte-server and airbyte-worker containers periodically, as mentioned in this [forum discussion](https://discuss.airbyte.io/t/2249). The Airbyte team is investigating a potential memory leak issue in the logging system, and you can track the progress on this issue [here](https://github.com/airbytehq/airbyte/issues/16652). Additionally, you can customize the memory settings for Java connectors by setting the

JOB_MAIN_CONTAINER_MEMORY_REQUEST

and

JOB_MAIN_CONTAINER_MEMORY_LIMIT

environment variables to custom values ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)). If you still face issues, consider sharing more details about your setup, such as the connectors you're using and the size of your sync, so we can provide more specific guidance.

I have 16 gb ram, out of 13 gb is available for airbyte 4 cpu cores are available I gave 3 gb to JOB_MAIN_CONTAINER_MEMORY_LIMIT , so source and destination are within this limit airbyte-server takes about 1-2 gb all these are fine but airbyte-worker is taking 8 gb and increasing constantly @kapa.ai

A message from kapa.ai

It seems like you might be experiencing a memory leak issue with the Airbyte worker. The Airbyte team is aware of a possible issue with memory leaks in the logging system [as mentioned in this forum discussion](https://discuss.airbyte.io/t/2249). Until a solution is merged, the suggested workaround is to restart the

airbyte-server

and

airbyte-worker

containers. You can track the progress of this issue on GitHub: [https://github.com/airbytehq/airbyte/issues/16652](https://github.com/airbytehq/airbyte/issues/16652).

This happens in every sync @kapa.ai I cannot restart between sync none of the sync is getting finished due to this issuue

A message from kapa.ai

I understand that restarting between syncs is not a feasible solution for you. Unfortunately, based on the information provided in the extracts, there isn't a direct solution to this issue yet. The Airbyte team is actively investigating the memory leak issue, and you can track the progress on GitHub: [https://github.com/airbytehq/airbyte/issues/16652](https://github.com/airbytehq/airbyte/issues/16652). In the meantime, you can try monitoring the memory usage and restarting the containers when necessary, or consider using a different version of Airbyte that might not have this issue. Keep an eye on the GitHub issue for updates and potential fixes.

@kapa.ai which version dont have this issue ?

A message from kapa.ai

New message text here