guys, am facing a critical issue in our aws ec2. s...
# random
t
guys, am facing a critical issue in our aws ec2. somehow it says storage space full and our app is now completely dead in prod. think it may be due to a docker container. can anybody pls help over a quick call? happy to pay consulting fee if required
👀 1
c
what was the attached volume size? if it was 8gb, i'd suggest updating the volume to >= 30gb. you can also do a
docker system prune -a
to cleanup unused containers/volumes for a quick solution
also, add some monitoring alerts for cpu/ram/storage/network etc.
t
@clever-advantage-29037 yes youre right, it was 8gb. ive just increased the volume size to 10gb and did resize2fs. its working now thankfully
but what im wondering is how all 8gb was even used. i have hasura and my db inside it, but i dont think that wouldve been 8gb... i have a feeling it may have been due to some dangling container or cache image. im new to all this, so this has been a nightmare to address in prod 😂
and yes, will definitely setup alerts
would you recommend i extend beyond 10gb?
g
I'd also recommend moving out your DB or most importantly keeping a regular automated (and tested) backup for the data. Docker sometimes takes up a lot of space for images due to its filesystem quirks. You've only added 2 GB of extra space from a production down event. It'll be better to atleast double the storage to get some time to figure out root cause and a proper solution.
t
yes, im gonna double the size. i just checked it and found that the utilization had increased from 81% to 83% in the same volume in last few hrs 😱 something is very wrong in the amount of space its eating and the pace.
c
could be excessive logs too?
t
could be... honestly have no idea how to go about fixing that as im new to this. these are my current volumes and storage
c
try
sudo du -hd 1 /
to see which dir is actually consuming most space
t
the biggest usage was /var/lib. inside that i tried to see top files and got this..:
c
yeah, just do
docker system prune -a
(also, what is the base image size?)