How can I free storage space on EC2 (with Airbyte) without deleting any valuable data?

Here are the main ways to free storage space on an EC2 running Airbyte without deleting valuable data: 1. Reduce Log Retention Period The most effective solution is to configure Airbyte to retain logs for fewer days. You can set the

TEMPORAL_HISTORY_RETENTION_IN_DAYS

environment variable in your

.env

or

values.yml

file: [log retention config; log sweeping]

env_vars:
  TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7

The default is 30 days. Setting it to a lower value (e.g., 7 days) will cause older logs to be cleaned up more frequently. Note that this feature was reimplemented in Airbyte v0.40.5+. [log sweeping] 2. Manually Delete Workspace Logs The main culprit for storage growth is typically the

/var/lib/docker/volumes/airbyte_workspace/_data

directory, which stores sync logs. You can safely delete old log files from this location. [storage issue] For Kubernetes deployments using MinIO, you can exec into the MinIO pod and delete logs under: [MinIO cleanup; hanging jobs fix] •

/storage/airbyte-storage/job-logging/workspace

•

/storage/airbyte-storage/workload/output

3. Clean Up Temporary Tables from Failed Syncs If you've had failed syncs (especially large ones), data may be stuck in

_airbyte_tmp_*

tables in your destination. These are not automatically deleted on failure and can consume significant space. You should manually clean these up from your destination database. [tmp tables] 4. Run

docker system prune

You can run

docker system prune

to remove unused Docker images, containers, and volumes. However, note that this alone may not solve the issue if the workspace logs are the primary cause. [disk space issue] ------------------------- Key takeaway: The most sustainable fix is setting

TEMPORAL_HISTORY_RETENTION_IN_DAYS

to a lower value so logs are automatically cleaned up more frequently going forward, combined with a one-time manual deletion of existing old logs. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

with linux commands please @kapa.ai

Based on the knowledge sources, here are the Linux commands you can use: 1. Check Docker Volume Usage

docker system df

2. Browse and Delete Workspace Logs Open a shell inside the

airbyte_workspace

Docker volume using BusyBox:

docker run -it --rm --volume airbyte_workspace:/data busybox

Then list the contents to identify old job folders:

ls /data/

Or run it directly from the host shell without entering the container:

docker run -it --rm --volume airbyte_workspace:/data busybox ls /data/9/2

[browsing logs] 3. Inspect a Specific Job's Log File

docker run -it --rm --volume airbyte_workspace:/data busybox cat /data/9/2/logs.log

[browsing logs] 4. Copy Files Out Before Deleting (Optional Safety Step)

docker cp airbyte-server:/tmp/workspace/9/2/catalog.json .

[browsing logs] 5. Prune Unused Docker Resources

docker system prune

[disk space issue] ------------------------- > Note: The knowledge sources don't provide explicit

rm

commands for deleting log folders directly. The recommended sustainable approach remains setting

TEMPORAL_HISTORY_RETENTION_IN_DAYS

in your

.env

file to a lower value (e.g., 7) so Airbyte handles cleanup automatically. [log sweeping] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.