Where does Airbyte store its logs? It’s maxed out ...
# ask-community-for-troubleshooting
Where does Airbyte store its logs? It’s maxed out the storage on the EC2 cluster and I would like to clear them out
Hey @Phil Marius, hopefully this docs page will help! Let me know if anything is unclear so we can make it better 😄
Yeah had a look at that, it’s fully bricked the machine though so can’t pull any new docker images even. Do you have docs on logs best practices / how to delete logs?
I tried clearing volumes with
docker-compose down -v
and there’s no storage to do that even
I don't think we have any docs on this, but it's a good idea for some (@abhi) @Phil Marius are you able to get into one of the containers via:
docker exec -it <container name/id> /bin/sh
Runs, but nothing happens
Copy code
$ docker exec -it <id> /bin/sh
Blank line
Hey @Phil Marius you can go to
and delete all the old jobs or inside which you will find the logs file you can delete that. I hope this helps To get into scheduler 1. docker ps - you will get all the containers 2. docker exec -it <id of scheduler> bash 3. cd /tmp/workspace
I cannot
exec -it
into the image
Copy code
$ docker exec -it <id> /bin/sh
OCI runtime exec failed: write /tmp/runc-process448723027: no space left on device: unknown
docker-compose down
Ha then for time being can you increase the size you can do this by going to volumes in aws dashboard you can increase probably by 8gb
this 1
Was hoping to avoid that but fair enough will give it a shot. Is there anything on the roadmap to have some way of dealing with logs?
As far as I remember the old logs will automatically get deleted which is already implemented by airbyte. But I understand this would have happened before that logic kicked it as that logic is around the number of jobs. In ideally scenario I see is moving logs to s3 which is there in k8s implementation but not sure in ec2 thing.
Ah kk yeah we’ve thought about rolling with the k8s setup in the future but we’re lacking k8s knowledge in the team atm
Cheers for the update though!
@Phil Marius you can do
docker airbyte-scheduler stop
echo "" > $(docker inspect --format='{{.LogPath}}' airbyte-scheduler
docker airbyte-scheduler start
with that you should have space to execute the commands.
Thanks @[DEPRECATED] Marcos Marx that was helpful. Although the specific commands that worked for me were:
Copy code
docker stop airbyte-scheduler
sudo echo "" > $(docker inspect --format='{{.LogPath}}' airbyte-scheduler)
docker start airbyte-scheduler
Then I could follow the the usual busybox instructions to delete the old logs from /data/ in the container.