https://linen.dev logo
Title
y

Yusuf Mirkar

05/25/2023, 10:15 AM
airbyte-worker is taking too much memory, it just increases to 8 gb until it uses all
u

user

05/25/2023, 10:15 AM
A message from kapa.ai
u

user

05/25/2023, 10:17 AM
It seems you're experiencing an issue with increasing memory usage in the Airbyte worker. This could be related to the Source worker, which reads up to 10,000 records in memory and can cause high memory usage for large row sizes in database sources ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)). One possible workaround is to restart the airbyte-server and airbyte-worker containers periodically, as mentioned in this [forum discussion](https://discuss.airbyte.io/t/2249). The Airbyte team is investigating a potential memory leak issue in the logging system, and you can track the progress on this issue [here](https://github.com/airbytehq/airbyte/issues/16652). Additionally, you can customize the memory settings for Java connectors by setting the
JOB_MAIN_CONTAINER_MEMORY_REQUEST
and
JOB_MAIN_CONTAINER_MEMORY_LIMIT
environment variables to custom values ([source](https://docs.airbyte.com/operator-guides/scaling-airbyte#memory)). If you still face issues, consider sharing more details about your setup, such as the connectors you're using and the size of your sync, so we can provide more specific guidance.
y

Yusuf Mirkar

05/25/2023, 10:18 AM
I have 16 gb ram, out of 13 gb is available for airbyte 4 cpu cores are available I gave 3 gb to JOB_MAIN_CONTAINER_MEMORY_LIMIT , so source and destination are within this limit airbyte-server takes about 1-2 gb all these are fine but airbyte-worker is taking 8 gb and increasing constantly @kapa.ai
u

user

05/25/2023, 10:18 AM
A message from kapa.ai
u

user

05/25/2023, 10:19 AM
It seems like you might be experiencing a memory leak issue with the Airbyte worker. The Airbyte team is aware of a possible issue with memory leaks in the logging system [as mentioned in this forum discussion](https://discuss.airbyte.io/t/2249). Until a solution is merged, the suggested workaround is to restart the
airbyte-server
and
airbyte-worker
containers. You can track the progress of this issue on GitHub: [https://github.com/airbytehq/airbyte/issues/16652](https://github.com/airbytehq/airbyte/issues/16652).
y

Yusuf Mirkar

05/25/2023, 10:20 AM
This happens in every sync @kapa.ai I cannot restart between sync none of the sync is getting finished due to this issuue
u

user

05/25/2023, 10:20 AM
A message from kapa.ai
u

user

05/25/2023, 10:21 AM
I understand that restarting between syncs is not a feasible solution for you. Unfortunately, based on the information provided in the extracts, there isn't a direct solution to this issue yet. The Airbyte team is actively investigating the memory leak issue, and you can track the progress on GitHub: [https://github.com/airbytehq/airbyte/issues/16652](https://github.com/airbytehq/airbyte/issues/16652). In the meantime, you can try monitoring the memory usage and restarting the containers when necessary, or consider using a different version of Airbyte that might not have this issue. Keep an eye on the GitHub issue for updates and potential fixes.
y

Yusuf Mirkar

05/25/2023, 10:21 AM
@kapa.ai which version dont have this issue ?
u

user

05/25/2023, 10:21 AM
A message from kapa.ai
u

user

05/25/2023, 10:22 AM
New message text here