Where does Airbyte store its logs? It’s maxed out the storage on the EC2 cluster and I would like to clear them out

Hey @Phil Marius, hopefully this docs page will help! Let me know if anything is unclear so we can make it better 😄

Yeah had a look at that, it’s fully bricked the machine though so can’t pull any new docker images even. Do you have docs on logs best practices / how to delete logs?

I don't think we have any docs on this, but it's a good idea for some (@abhi) @Phil Marius are you able to get into one of the containers via:

docker exec -it <container name/id> /bin/sh

Runs, but nothing happens

$ docker exec -it <id> /bin/sh

Hey @Phil Marius you can go to

/tmp/workspace

in

scheduler

and delete all the old jobs or inside which you will find the logs file you can delete that. I hope this helps To get into scheduler 1. docker ps - you will get all the containers 2. docker exec -it <id of scheduler> bash 3. cd /tmp/workspace

I cannot

exec -it

into the image

$ docker exec -it <id> /bin/sh
OCI runtime exec failed: write /tmp/runc-process448723027: no space left on device: unknown

Ha then for time being can you increase the size you can do this by going to volumes in aws dashboard you can increase probably by 8gb

Was hoping to avoid that but fair enough will give it a shot. Is there anything on the roadmap to have some way of dealing with logs?

As far as I remember the old logs will automatically get deleted which is already implemented by airbyte. But I understand this would have happened before that logic kicked it as that logic is around the number of jobs. In ideally scenario I see is moving logs to s3 which is there in k8s implementation but not sure in ec2 thing.

Ah kk yeah we’ve thought about rolling with the k8s setup in the future but we’re lacking k8s knowledge in the team atm

@Phil Marius you can do

docker airbyte-scheduler stop

then

echo "" > $(docker inspect --format='{{.LogPath}}' airbyte-scheduler

and

docker airbyte-scheduler start

with that you should have space to execute the commands.

Thanks @[DEPRECATED] Marcos Marx that was helpful. Although the specific commands that worked for me were:

docker stop airbyte-scheduler
sudo echo "" > $(docker inspect --format='{{.LogPath}}' airbyte-scheduler)
docker start airbyte-scheduler

Then I could follow the the usual busybox instructions to delete the old logs from /data/ in the container.

Awesome