Airbyte is an open-source data integration engine that helps you consolidate your data in your data warehouses, lakes and databases.

Airbyte

Does anyone have a solution to the storage disk being maxed out after a couple of days?

whats taking up the space logs? local destination?

<https://discuss.airbyte.io/t/airbyte-rotation-logs/332/2>

```r 100+ unused docker images (perhaps related to old connectors?) that were eating up 20GB. docker prune image -a cleaned things up, in case this thread is of use to anyone else who finds it in the future.```

Thanks for the link but that didnt work for me

32GB
Yeah its writing to the local destination

then im not sure there is much you can do, its based on the data you are pushing.. other than push less data or prune it after it gets synced
what sync mode are you using?
if you are using an append mode, it will keep growing for example

yeah so it will grow forever then, adding new data each replication if there is new data

<https://docs.airbyte.com/understanding-airbyte/connections/>
explains the 4 options.
if the table is small, full overwrite may be a better ideas

incremental append will grow slower than full append

This blog also dives deep into it:
<https://airbyte.com/blog/understanding-data-replication-modes>
probably a little easier to digest than the documentation

I understand the replication modes.
The issue is that the log files occupy most of the storage disk
Which is why i want to clear the logs for previous days in the workspace

how much data are you replicating and how often?
I replicating 500million rows last night and only have a few gigs of logs

<@U01MMSDJGC9> turned this thread into Zendesk ticket <https://airbyte7538.zendesk.com/agent/tickets/2907 | 2907> to ensure timely resolution!

Hey <@U02HR3NTRNV>, thanks for the question. How much storage have you allocated, what connectors are you using?

This is a command that another user shared for deleting the logs manually on their EC2 instance. Although as Robert said, if you’re using an append sync, the logs will get bigger and you’ll have to run this command every few days.

`find /var/lib/docker/volumes/airbyte_workspace/_data/ -maxdepth 1 -regex '/var/lib/docker/volumes/airbyte_workspace/_data/[0-9]+' -mtime +10 | xargs rm -rf`