https://linen.dev logo
r

Raj

02/07/2022, 5:30 PM
Hi all, After the upgrade to 0.35.x we have observed airbyte is using huge amount of ram. In the below screenshots with no jobs running, airbyte is utilizing 10-11GB RAM on an average. is this expected ?? We had to upgrade from a 16GB system to 32GB as 16GB was no more sufficient
Hey these are a couple things that would make this much easier to dive into: 1. Having them run 
docker stats
 so we can see how much each container is using. 2. Also knowing how many connections they have in total and which connectors they are using.
m

Mohamed Magdy

02/08/2022, 6:48 PM
@Harshith (Airbyte)
When i took the screenshot we have 0 active connections
Shared this across with team.
Hey do you deploy this with docker-compose and do you set any resource limits ?
yes, we use docker-compose and we did not set any resource limits
w

Will Sargent

02/09/2022, 6:21 PM
@Harshith (Airbyte) currently two active connections and ram usage of airbyte server is almost 15GB. I think this is critical issue. Can we raise this with a high priority ?
@Davin Chia (Airbyte) can you help here?
this seems to be working ok on my local:
what happens if you limit the RAM given to the docker agent <10 GB, does the airbyte server OOM?
@Davin Chia (Airbyte) It is working for us but the problems is we recently upgraded to 0.35.22. A 32GB system is hardly managing to run 3-4 parallel connections. Before upgrading we have 0.29.22 which was working like charm even with 16gb server. The issue is resource usage. It has become to costly if airbyte is using this much resources.
right what makes you think it actually requires that much RAM? did it OOM previously before you moved to a larger instance?
Yes, when we are using 16gb servers with upraded Airbyte, the server is crashing after the memory is full. Connectors start giving DEADLINE_EXCEEDED error
We used 0.29.22 version on 16GB server, we were able to comfortable handle up to 10 - 20 parallel connections sometime.
what was the error message when the server crashed?
We cannot ssh into ec2 as the memory is full.. we have to restart the system before doing anything
can you limit the docker agent memory so the base instance isn’t affected? and see if you can retrieve an error message?
I can’t think of anything that would cause this error off the top of my head. we’ve been running Airbyte in our own Cloud and haven’t noticed any memory issues between 0.29 and 0.35
I am not sure if this helps by default memory is limit to 75%
can you please download and attach the server logs here
maybe there will be some clues in there
Its crashed on 5th Feb and 7th feb later we upgraded to the server capacity
@Davin Chia (Airbyte) Any clue ??
t

Thanh Le

02/13/2022, 5:12 PM
@Davin Chia (Airbyte) can you please help us with this. It is costing us a lots to run this server
n

Naveen Sai Patnana

02/16/2022, 3:04 AM
It’s possible to you disable all connections and restart Airbyte? Check if only the server/app the same resources are requested?
4 Views