Hi all,
After the upgrade to 0.35.x we have obser...
# contributing-to-airbyter
Raj
02/07/2022, 5:30 PMHi all,
After the upgrade to 0.35.x we have observed airbyte is using huge amount of ram. In the below screenshots with no jobs running, airbyte is utilizing 10-11GB RAM on an average. is this expected ?? We had to upgrade from a 16GB system to 32GB as 16GB was no more sufficient
Raj
02/08/2022, 1:58 PMHey these are a couple things that would make this much easier to dive into:
1. Having them run
docker statsm
Mohamed Magdy
02/08/2022, 6:48 PM@Harshith (Airbyte)
Mohamed Magdy
02/08/2022, 6:48 PMWhen i took the screenshot we have 0 active connections
Mohamed Magdy
02/09/2022, 8:46 AMShared this across with team.
Mohamed Magdy
02/09/2022, 9:13 AMHey do you deploy this with docker-compose and do you set any resource limits ?
Mohamed Magdy
02/09/2022, 9:25 AMyes, we use docker-compose and we did not set any resource limits
w
Will Sargent
02/09/2022, 6:21 PM@Harshith (Airbyte) currently two active connections and ram usage of airbyte server is almost 15GB. I think this is critical issue. Can we raise this with a high priority ?
Will Sargent
02/09/2022, 8:26 PM@Davin Chia (Airbyte) can you help here?
Will Sargent
02/10/2022, 9:34 AMthis seems to be working ok on my local:
Will Sargent
02/10/2022, 9:34 AMwhat happens if you limit the RAM given to the docker agent <10 GB, does the airbyte server OOM?
Will Sargent
02/10/2022, 11:04 AM@Davin Chia (Airbyte) It is working for us but the problems is we recently upgraded to 0.35.22. A 32GB system is hardly managing to run 3-4 parallel connections. Before upgrading we have 0.29.22 which was working like charm even with 16gb server. The issue is resource usage. It has become to costly if airbyte is using this much resources.
Will Sargent
02/10/2022, 11:05 AMright what makes you think it actually requires that much RAM? did it OOM previously before you moved to a larger instance?
Will Sargent
02/10/2022, 11:08 AMYes, when we are using 16gb servers with upraded Airbyte, the server is crashing after the memory is full. Connectors start giving DEADLINE_EXCEEDED error
Will Sargent
02/10/2022, 11:10 AMWe used 0.29.22 version on 16GB server, we were able to comfortable handle up to 10 - 20 parallel connections sometime.
Will Sargent
02/10/2022, 11:14 AMwhat was the error message when the server crashed?
Will Sargent
02/10/2022, 11:14 AMWe cannot ssh into ec2 as the memory is full.. we have to restart the system before doing anything
Will Sargent
02/10/2022, 11:15 AMcan you limit the docker agent memory so the base instance isn’t affected? and see if you can retrieve an error message?
Will Sargent
02/10/2022, 11:16 AMI can’t think of anything that would cause this error off the top of my head. we’ve been running Airbyte in our own Cloud and haven’t noticed any memory issues between 0.29 and 0.35
Will Sargent
02/10/2022, 11:20 AMI am not sure if this helps by default memory is limit to 75%
Will Sargent
02/10/2022, 11:22 AMhttps://airbytehq.slack.com/archives/C019WEENQRM/p1644250651042229 Please check this also. Same issues
Will Sargent
02/10/2022, 11:25 AMcan you please download and attach the server logs here
Will Sargent
02/10/2022, 11:26 AMmaybe there will be some clues in there
Will Sargent
02/10/2022, 11:47 AMIts crashed on 5th Feb and 7th feb later we upgraded to the server capacity
Will Sargent
02/11/2022, 6:18 AM@Davin Chia (Airbyte) Any clue ??
t
Thanh Le
02/13/2022, 5:12 PM@Davin Chia (Airbyte) can you please help us with this. It is costing us a lots to run this server
n
Naveen Sai Patnana
02/16/2022, 3:04 AMIt’s possible to you disable all connections and restart Airbyte? Check if only the server/app the same resources are requested?
4 Views