@kapa.ai I have 200GB disk on aws ec2, and disk is full now. I need to clear unused data, logs etc that will not impact my active connections. How can i do that

Here are the safe ways to free up disk space on your Airbyte EC2 instance without impacting active connections: 1. Clean Up Workspace/Log Files The main culprit for disk usage is typically the workspace volume. You can manually delete log folders: • Docker deployments: The main source of growth is usually

/var/lib/docker/volumes/airbyte_workspace/_data

. You can purge old log folders from there. [storage issue] • Kubernetes deployments: SSH/exec into the MinIO pod and delete logs under: ◦

/storage/airbyte-storage/job-logging/workspace

◦

/storage/airbyte-storage/workload/output

[sync jobs hanging] 2. Configure Automatic Log Retention To prevent the disk from filling up again, set the

TEMPORAL_HISTORY_RETENTION_IN_DAYS

environment variable in your

.env

or

values.yml

file. For example, to retain logs for only 7 days:

env_vars:
  TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7

The default is 30 days. Reducing this value will significantly reduce disk usage over time. [log sweeping; storage issue] 3. Clean Up _airbyte_tmp_* Tables in the Destination If you've had failed syncs, temporary tables (

_airbyte_tmp_*

) may have been left behind in your destination database. These are not automatically deleted on failure and can consume significant space. You can manually drop them from your destination. [S3 failed attempts] What to Avoid • Do not use the "Clear data" feature in the Airbyte UI unless you intentionally want to wipe synced data from your destination tables — this will cause data downtime and is not reversible. [clearing data] These steps should free up significant disk space without affecting your active connections or their configurations. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

Deleting everything in this folder will not impact my active connections and jobs right @kapa.ai

Based on the knowledge sources available, I can provide some clarity, but with important caveats: What the workspace folder contains: The

airbyte_workspace

volume (at

/var/lib/docker/volumes/airbyte_workspace/_data

) stores sync job artifacts such as

catalog.json

,

logs.log

,

tap_config.json

, and

target_config.json

— these are temporary working files and logs for each sync attempt. [browsing logs] What is safe to say: • Airbyte itself schedules periodic jobs to delete these files. By default, logs are deleted when they exceed 5GB or after 60 days, which implies these files are considered expendable. [log retention] • Users in the community have manually deleted folders from this workspace to free up space without reporting loss of connection configurations. [workspace space; storage issue] Important limitation: The knowledge sources do not explicitly confirm that deleting everything in the workspace folder is completely safe for all active connections and in-progress jobs. If a sync job is currently running, deleting its working directory could interfere with that job. My recommendation: • Ensure no syncs are actively running before deleting. • Your connection configurations, credentials, and sync history metadata are stored in the Airbyte database (

airbyte_db

), not in the workspace volume, so those should not be affected. • Consider configuring the retention environment variables (

MAXIMUM_WORKSPACE_RETENTION_DAYS

,

MAXIMUM_WORKSPACE_SIZE_MB

) to let Airbyte manage cleanup automatically going forward. [log retention] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.

where is .env located for

TEMPORAL_HISTORY_RETENTION_IN_DAYS

@kapa.ai

The

.env

file is located in the root directory of your Airbyte installation. If you followed the standard installation instructions, it would be in a directory named `airbyte`:

cd airbyte
nano .env

Then add or update the variable:

TEMPORAL_HISTORY_RETENTION_IN_DAYS=7

After editing, restart Airbyte for the changes to take effect:

docker-compose down
docker-compose up

[configuring Airbyte] The default value is 30 days, so setting it to a lower number (e.g., 7) will help keep disk usage under control going forward. [data retention] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.