@kapa.ai it seems like the storage for my airbyte instance is slowly filling up. I have airbyte running on EC2. Is there a process to clean that data before it causes problems and bogs down my instance?

Yes, there are processes to clean up storage for your Airbyte instance running on EC2 before it causes problems. Here are some key points and recommendations based on the information provided: 1. Log Retention: Airbyte schedules periodic jobs to delete logs from syncs. By default, these logs are deleted every month or when their size exceeds 5GB. You can adjust this behavior by setting environment variables: - `TEMPORAL_HISTORY_RETENTION_IN_DAYS`: This configures how long to retain logs. For example, setting it to 1 will retain logs for only 1 day. [Forum Post: Airbyte log sweeping](https://discuss.airbyte.io/t/2714) 2. Manual Cleanup: For immediate relief, you can manually clean up the Minio storage, which stores logs: - If running on Kubernetes, you can exec into the Minio pod and delete files in

/storage/airbyte-dev-logs/job-logging/workspace/

- You might also want to delete files in

/storage/airbyte-dev-logs/job-logging/workspace/

[Forum Post: How to clean up airbyte-minio and what's stored in minio?](https://discuss.airbyte.io/t/3101) 3. Workspace Cleanup: The

/var/lib/docker/volumes/airbyte_workspace/_data

folder often consumes a large amount of space. While there isn't a built-in automatic cleanup for this, you can manually purge this folder. However, be cautious as this contains important data for Airbyte operations. [Github Issue: Airbyte slowness issue in Open source Environment](https://github.com/airbytehq/airbyte/issues/29754) 4. Increase Storage: If cleaning up doesn't provide enough relief, consider increasing your EC2 instance's storage capacity. 5. Regular Monitoring: Implement regular monitoring of your storage usage to preemptively address space issues before they impact performance. Remember to be cautious when manually deleting data, as it may affect Airbyte's functionality. Always ensure you have proper backups before performing any cleanup operations.