https://linen.dev logo
Join Slack
Powered by
<@U04S8PLJWMT> How can I purge minio on an abctl d...
# replication-ask-ai
j
@kapa.ai How can I purge minio on an abctl deployment ?
k
The knowledge sources don’t describe an 
abctl
-specific command to purge MinIO, but they do show how others clean it up on Kubernetes-based deployments like the one 
abctl
creates. On an 
abctl
(local K8s) deployment, MinIO is just another pod with a mounted PVC. To purge it you have two main options: ### 1. Delete logs/data inside MinIO (keep PVC) Exec into the 
airbyte-minio
pod and remove the log/output directories:
Copy code
bash kubectl exec -it deploy/airbyte-minio -- sh # inside the pod: rm -rf /storage/airbyte-storage/job-logging/workspace/* rm -rf /storage/airbyte-storage/workload/output/* exit
Users report that deleting these paths frees space and resolves “storage full” / hanging jobs issues while keeping MinIO itself. [[MinIO storage full](https://discuss.airbyte.io/t/1857); [All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] Depending on your chart/version, the base path may differ (e.g. 
/storage/airbyte-dev-logs/job-logging/workspace/
), but the pattern is the same: clean the 
job-logging/workspace
(and optionally 
workload/output
) directories. [[Clean up minio](https://discuss.airbyte.io/t/3101); [All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] ### 2. Delete and recreate MinIO storage (PVC) Some users instead delete the MinIO PVC so Kubernetes recreates empty storage: 1. Identify the PVC:
Copy code
bash kubectl get pvc | grep minio
2. Delete the PVC (this wipes all MinIO data, including logs):
Copy code
bash kubectl delete pvc <airbyte-minio-pvc-name>
Kubernetes will re-provision it according to the Helm chart, and users report this fixes stuck jobs caused by full MinIO. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812); [Clean up minio](https://discuss.airbyte.io/t/3101)] ### Optional: Increase MinIO disk / retention To avoid needing to purge frequently: - Increase the MinIO volume size in your 
values.yaml
, for example:
Copy code
yaml minio:   storage:     volumeClaimValue: 2Gi
[[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] - Reduce Temporal history retention to keep fewer logs:
Copy code
yaml env_vars:   TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7
[[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] If you need exact paths or commands for your specific 
abctl
version, the sources don’t provide more detail, so you may need to inspect the MinIO pod (
kubectl exec
and 
ls /storage
) to confirm the directory structure.
j
@kapa.ai how can I Exec into the 
airbyte-minio
pod
k
A message from kapa.ai
To exec into the 
airbyte-minio
pod in an 
abctl
(kind/Kubernetes) deployment, you can use 
kubectl exec
the same way as for any other Airbyte pod. 1. First, list pods in the 
airbyte-abctl
namespace to get the exact MinIO pod name:
Copy code
bash kubectl get pods -n airbyte-abctl
You should see something like 
airbyte-minio-0
in the list. [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development)] 2. Exec into that pod:
Copy code
bash kubectl exec -it airbyte-minio-0 -n airbyte-abctl -- sh
This opens a shell inside the MinIO container, where you can run normal Linux commands. The pattern is the same as the documented example for other Airbyte pods on Kubernetes. [[Browsing logs](https://docs.airbyte.com/platform/2.0/operator-guides/browsing-output-logs#browsing-on-kubernetes)]
j
@kapa.ai Since I only use docker and abctl, I get the 
kubectl: command not found
error
k
A message from kapa.ai
The knowledge sources don’t describe how to manage MinIO without 
kubectl
, and 
abctl
itself doesn’t expose a built‑in “exec into pod” or “purge MinIO” command. They assume you have 
kubectl
installed alongside 
abctl
to interact with the kind cluster. [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development); [abctl troubleshooting](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] From the docs: - 
abctl
creates a kind (Kubernetes-in-Docker) cluster and deploys Airbyte into it. [[abctl overview](https://docs.airbyte.com/platform/deploying-airbyte/abctl)] - To “view logs, debug issues, and manage your Airbyte deployment, you should install … kubectl for interacting with the Kubernetes cluster.” [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development)] - The troubleshooting guide explicitly shows using 
kubectl
with the 
abctl
kubeconfig to interact with pods/resources. [[abctl troubleshooting](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] Given that, the documented path to purge MinIO on an 
abctl
deployment is: 1. Install 
kubectl
 (even though you “only use Docker and abctl”, 
kubectl
is required to reach the MinIO pod in the kind cluster). 2. Point 
kubectl
at the 
abctl
cluster (either via 
kind export kubeconfig -n airbyte-abctl
or the explicit 
--kubeconfig ~/.airbyte/abctl/abctl.kubeconfig
flag). [[Developing locally](https://docs.airbyte.com/platform/2.0/contributing-to-airbyte/developing-locally#using-abctl-for-airbyte-development); [abctl troubleshooting](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] 3. Use 
kubectl exec
into 
airbyte-minio-0
and delete the log directories as described earlier. The sources do not document an alternative that avoids installing 
kubectl
(for example, directly `docker exec`‑ing into the MinIO container created by kind), so I can’t reliably recommend such a method based on the provided material.
j
How to clean up the disk then ? @kapa.ai
k
A message from kapa.ai
You’ll need to clean up the MinIO storage inside the Kubernetes cluster that 
abctl
creates. The docs assume you install and use 
kubectl
alongside 
abctl
for this. There is no 
abctl
command to purge MinIO directly. [[abctl troubleshooting](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] ### 1. Install and point 
kubectl
to the abctl cluster 1. Install 
kubectl
for your OS (per Kubernetes docs). 2. Use the kubeconfig that 
abctl
created:
Copy code
bash export KUBECONFIG="$HOME/.airbyte/abctl/abctl.kubeconfig"
You can now run 
kubectl
against the 
airbyte-abctl
cluster. [[abctl troubleshooting](https://docs.airbyte.com/platform/deploying-airbyte/troubleshoot-deploy#faq)] ### 2. Exec into the MinIO pod and delete logs List pods to find MinIO:
Copy code
bash kubectl get pods -n airbyte-abctl
You should see something like 
airbyte-minio-0
. Exec into it:
Copy code
bash kubectl exec -it airbyte-minio-0 -n airbyte-abctl -- sh
Inside the pod, delete the log/output data that fills the disk. Users report these paths as the main culprits:
Copy code
bash rm -rf /storage/airbyte-storage/job-logging/workspace/* rm -rf /storage/airbyte-storage/workload/output/*
On some setups the base path is 
/storage/airbyte-dev-logs
instead of 
/storage/airbyte-storage
, but the important subpaths are 
job-logging/workspace
and (optionally) 
workload/output
. [[S3 Minio storage full](https://discuss.airbyte.io/t/1857); [How to clean up minio](https://discuss.airbyte.io/t/3101); [All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] Exit the shell when done. ### 3. (Alternative) Delete the MinIO PVC If you want to wipe MinIO completely:
Copy code
bash kubectl get pvc -n airbyte-abctl | grep minio kubectl delete pvc <minio-pvc-name> -n airbyte-abctl
Kubernetes will recreate an empty volume according to the Helm chart. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] ### 4. Prevent it from filling again (optional) - Increase MinIO disk size in your Helm values (e.g. 
volumeClaimValue
). - Reduce Temporal history retention:
Copy code
yaml env_vars:   TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7
This reduces how much history/log data is kept and helps avoid MinIO filling up. [[All sync jobs hanging](https://github.com/airbytehq/airbyte/issues/55812)] The knowledge sources don’t document any way to clean up MinIO disk space without using 
kubectl
, so installing 
kubectl
and using the steps above is the supported path.
3 Views
Open in Slack
PreviousNext
Powered by