Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Not sure if this has been asked before, but is there a best practice way of maintaining the contents of the airbyte_workspace folder? I see each run of my jobs creating new scripts and log files, which is great, but over time, it's now chewing up all of the disk space on my small instance. I don't want to break something by wiping out the contents of the folder, so wondered if there was a proper way of maintaining the contents of this folder, maybe only allowing airbyte to keep say 7 days worth of logs, or something similar?

There is a sweeper process that runs in the background to do cleanup

You can also override some values in the `.env` file if you want to customize the behavior.

`MINIMUM_WORKSPACE_RETENTION_DAYS` ,
`MAXIMUM_WORKSPACE_RETENTION_DAYS` , and `MAXIMUM_WORKSPACE_SIZE_MB` are the relevant constants

the way it works is it never deletes anything that is more recent than the `MINIMUM_WORKSPACE_RETENTION_DAYS`

It always deletes things older than `MAXIMUM_WORKSPACE_RETENTION_DAYS`

If it’s between the two, it only deletes files if the workspace size exceeds `MAXIMUM_WORKSPACE_SIZE_MB`

The default minimum is 1 day, the default maximum is 60 days, and the default maximum size is 5g

If you want to limit to 7 days you can just `docker-compose down`, set `MAXIMUM_WORKSPACE_RETENTION_DAYS` to 7 in the `.env` file, and `docker-compose up`. The next run of the job sweeper should clean up old files soon after.

<@U01ADKHBJKC> we should put that in the FAQ?

Does it get asked a lot? :slightly_smiling_face: Odd, mine had grown to about 15GB in size, and I'm pretty certain I'd gone over the 60 day period too based on what I could see, but couldn't be completely sure about that. I'll have a go with the MAXIMUM_WORKSPACE_RETENTION_DAYS though, sounds perfect.

I did try looking through docs etc. first before asking but the closest I could find was how to browse the AirByte_Workspace folder, which was actually really helpful in working out where the space on my tiny server was disappearing to. :smile:

Thanks for your help though <@U01ADKHBJKC> and <@U01A81VHD45>

I think it’s been asked at least one other time. It’s worth making this sort of thing searchable. For every person that asks a question/runs into a problem there’s probably at least a few people out there that won’t ask on Slack, so having better docs is critical for us.

You should be able to see `Deleting:  ...some path...` in the scheduler logs for each file it deletes by the way.