<@U04S8PLJWMT> what can i do if my disk has no fre...
# ask-ai
m
@kapa.ai what can i do if my disk has no free space left?
k
When Airbyte reports “no space left on device”, there are a few common culprits and fixes, depending on your deployment. ### 1. MinIO / S3-compatible storage is full If you use the bundled
airbyte-minio
(Kubernetes/OSS Helm): - MinIO mainly stores job logs and related outputs. [[Minio cleanup](https://discuss.airbyte.io/t/3101); [S3 Minio full](https://discuss.airbyte.io/t/1857)] - You can safely delete old logs inside the MinIO volume; examples from running pods: - Older path name:
/storage/airbyte-dev-logs/job-logging/workspace/
[[Minio cleanup](https://discuss.airbyte.io/t/3101)] - Newer path name:
/storage/airbyte-storage/job-logging/workspace
and possibly
/storage/airbyte-storage/workload/output
[[Syncs hanging](https://github.com/airbytehq/airbyte/issues/55812)] After deleting logs/outputs, MinIO frees disk and syncs can run again. To avoid recurrence, reduce how long Temporal keeps workflow history (this directly affects log retention in workspace):
Copy code
yaml env_vars:   TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7
Setting this env var (e.g., in your Helm values) shortens history and reduces MinIO usage. [[Syncs hanging](https://github.com/airbytehq/airbyte/issues/55812); [Log sweeping](https://discuss.airbyte.io/t/2714)] If you see an error like “Storage backend has reached its minimum free disk threshold” or
XMinioStorageFull
, that also indicates your MinIO disk is full and you must delete objects or increase the PVC size. [[S3 Minio full](https://discuss.airbyte.io/t/1857); [S3 + Minio issue](https://github.com/airbytehq/airbyte/issues/59197)] --- ### 2. Airbyte workspace logs on the host are too large (Docker / VM) Airbyte’s workspace (logs, job artifacts) can fill the host disk over time and cause: - Sync jobs failing - UI not loading - Commands like
docker logs
failing with “no space left on device” [[Syncs suddenly failing](https://discuss.airbyte.io/t/3303); [Airbyte rotation logs](https://discuss.airbyte.io/t/332)] Current behavior: - From v0.40.5+, Airbyte periodically cleans files from the workspace (default: every 30 days). [[Log sweeping](https://discuss.airbyte.io/t/2714)] - You can configure retention with:
Copy code
bash   TEMPORAL_HISTORY_RETENTION_IN_DAYS=1   # example: keep 1 day
This replaces the older
MINIMUM_WORKSPACE_RETENTION_DAYS
/
MAXIMUM_WORKSPACE_RETENTION_DAYS
/
MAXIMUM_WORKSPACE_SIZE_MB
mechanism, which no longer applies with the Temporal-based scheduler. [[Log sweeping](https://discuss.airbyte.io/t/2714)] If your disk is already full: - Manually delete old log folders under the Airbyte workspace directory (on many Docker installs this is something like
./data
where you cloned Airbyte), or, if using MinIO as above, clear logs from MinIO. --- ### 3. Destination database temp or tmp tables Sometimes the “no space left on device” is in the destination database, not the Airbyte host: - Example: Postgres / Aurora error
could not write to file "base/pgsql_tmp/..." : No space left on device
appears during normalization. [[No space normalization](https://discuss.airbyte.io/t/1572); [No space root cause](https://discuss.airbyte.io/t/1572)] In that case, you must free or increase storage on the destination DB side (e.g., clean up large temp files, increase storage allocated to the DB instance). [[No space root cause](https://discuss.airbyte.io/t/1572)] Also, failed syncs may leave large
_airbyte_tmp_*
tables in the destination; those are not auto-deleted when a sync fails and can consume space. You can drop these
_airbyte_tmp_*
tables manually in the destination DB. [[S3 source tmp tables](https://discuss.airbyte.io/t/2067)] --- ### 4. Jobs database or other containers filling disk A very large jobs database (Airbyte internal DB) can also fill the disk and bring
airbyte-db
down, which in turn breaks the whole instance. [[Syncs suddenly failing](https://discuss.airbyte.io/t/3303)] In that case you may need to: - Increase disk allocated to the host/volume used by `airbyte-db`; and/or - Manually prune job records in the jobs DB (not directly documented here, but this is what the referenced user did by truncating jobs history). [[Syncs suddenly failing](https://discuss.airbyte.io/t/3303)] --- ### 5. Summary of concrete actions Depending on where your space is exhausted: 1. MinIO PVC full - SSH/exec into MinIO pod and delete contents in: -
/storage/airbyte-storage/job-logging/workspace
-
/storage/airbyte-storage/workload/output
(or
/storage/airbyte-dev-logs/job-logging/workspace
on older setups). [[Minio cleanup](https://discuss.airbyte.io/t/3101); [Syncs hanging](https://github.com/airbytehq/airbyte/issues/55812)] - Optionally increase MinIO PVC size. - Set
TEMPORAL_HISTORY_RETENTION_IN_DAYS
to a lower value. 2. Host workspace full (Docker/VM) - Manually remove old log/job folders under the Airbyte workspace directory. - Upgrade to ≥0.40.5 and set
TEMPORAL_HISTORY_RETENTION_IN_DAYS
to control retention. [[Log sweeping](https://discuss.airbyte.io/t/2714)] 3. Destination DB out of space - Increase DB storage or clean up space (including temp files or
_airbyte_tmp_*
tables). [[No space root cause](https://discuss.airbyte.io/t/1572); [S3 source tmp tables](https://discuss.airbyte.io/t/2067)] If you share what deployment you’re using (Docker vs Kubernetes, and whether you see MinIO-related errors), I can point to the most relevant of these steps.
m
@kapa.ai is safe to a do a image prune?
k
A message from kapa.ai
It depends which prune command you mean, but in general: -
docker image prune
or
docker system prune
are safe
for Airbyte as long as you don’t remove Docker volumes, because all Airbyte configs/state are stored in volumes, not in images or running containers. [[Persistence GCP](https://discuss.airbyte.io/t/1381); [Compose up/down](https://discuss.airbyte.io/t/3836)] - A
docker-compose down
(or
docker compose down
) does not delete volumes by default, so stopping the stack and pruning unused images afterwards will not remove your connections or config. [[Compose up/down](https://discuss.airbyte.io/t/3836)] Be careful with any prune command that includes `--volumes`; the knowledge sources don’t describe that scenario, and deleting Airbyte volumes (e.g.
airbyte_db
,
airbyte_workspace
) would remove data.