Hi <@U04S8PLJWMT>, I migrated to V2 chrt. deployed...
# replication-ask-ai
p
Hi @kapa.ai, I migrated to V2 chrt. deployed helm chart. now im getting this issue on minio trying to bound to pvc. Warning FailedMount 99s (x15 over 16m) kubelet MountVolume.SetUp failed for volume "pvc-65f7d5a1-0579-4cc8-acbe-94b11a832491" : rpc error: code = InvalidArgument desc = nutanix: iSCSI portal info is missing for c6534d14-2bfb-4396-72e3-71a2c1227c2d, err: failed to get dataservices ip from cluster info, error: GetClusterInfo API returned unexpected type: <nil>
k
This looks like a Kubernetes/storage‑backend issue rather than something specific to the Airbyte V2 chart itself. From the available sources, similar symptoms (MinIO pod stuck Pending / FailedMount, PVC not binding) are always traced back to the underlying storage provisioner or StorageClass configuration, not to Airbyte configuration. [Minio pending; EKS PVC issues] In your case, the error is coming from the Nutanix CSI driver: >
rpc error: code = InvalidArgument desc = nutanix: iSCSI portal info is missing ... failed to get dataservices ip from cluster info
The Airbyte docs and issues do not provide Nutanix‑specific guidance or a way to configure iSCSI/portal details from the Helm chart, so I can’t give a chart‑level fix based on the knowledge sources. There is also an open issue where MinIO’s PVC cannot bind because the chart doesn’t expose storageClassName configuration, but that’s about choosing a StorageClass, not fixing a broken CSI driver. [Minio PVC issue] Given what’s documented: • The V2 chart expects a functioning dynamic provisioner / StorageClass for MinIO’s PVC to bind successfully. [Minio pending] • When the provisioner or underlying storage is misconfigured, the PVC stays Pending and MinIO never starts; the fix is on the Kubernetes/storage side, not in Airbyte. [Minio pending] So, based on the docs: 1. Verify that your Nutanix CSI driver and cluster storage configuration are healthy and that other PVCs using the same StorageClass can bind and mount successfully. 2. If you intend to use a different StorageClass for MinIO, note that there is currently a limitation in the chart around configuring MinIO’s PVC storage class, as reported in the open GitHub issue. [Minio PVC issue] The knowledge sources do not contain Nutanix‑specific steps or a direct workaround for this exact error, so you’ll need to resolve the Nutanix iSCSI / dataservices IP configuration at the Kubernetes/storage layer, then re‑deploy the chart. 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
I fixed that. now the jobs are stuck in sync mode and dont see any activity and cant see logs too. should i redploy? And also does minio has its separate airbyte dployment @kapa.ai
k
Good to hear the PVC issue is resolved! The symptoms you're now describing — jobs stuck in sync, no activity, no logs — are a known pattern after MinIO-related issues on Kubernetes deployments of Airbyte. Jobs Stuck / No Logs Based on the knowledge sources, this is very likely a MinIO storage space or state issue. When MinIO's PVC fills up or gets into a bad state, sync jobs hang indefinitely and logs become unavailable. [All syncs hanging] Here are the documented fixes, in order of preference: 1. Clear old logs from MinIO (least destructive): Go into the MinIO container and delete logs from: ◦
/storage/airbyte-storage/job-logging/workspace
◦
/storage/airbyte-storage/workload/output
2. Increase MinIO storage size in your `values.yaml`:
Copy code
minio:
     storage:
       volumeClaimValue: 2Gi
The default is only 500MB, which fills up quickly with frequent syncs. [All syncs hanging] 3. Optionally reduce Temporal history retention to prevent future overflow:
Copy code
env_vars:
     TEMPORAL_HISTORY_RETENTION_IN_DAYS: 7
The default is 30 days. [All syncs hanging] A full redeployment alone is unlikely to fix this if the underlying MinIO storage is the problem — restarting pods without clearing the state has been reported to not resolve the hanging jobs. [All syncs hanging] Does MinIO Have Its Own Deployment? Yes — MinIO is deployed as a separate pod within the Airbyte namespace as part of the Helm chart (when enabled). It has its own PVC for storage. You can disable it and use external S3 instead by setting
minio.enabled: false
and configuring S3 credentials in
values.yaml
. [Minio pending] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
i dont see anything for minio if check kubetcl get deployment @kapa.ai
k
That's expected — MinIO in Airbyte's Helm chart is deployed as a StatefulSet, not a Deployment. So you won't see it under
kubectl get deployment
. Try these commands instead:
Copy code
kubectl get statefulset -n airbyte
kubectl get pods -n airbyte
kubectl get pvc -n airbyte
You should see
airbyte-minio-0
as a pod and
airbyte-minio-pv-claim-airbyte-minio-0
as its PVC. [Minio PVC issue; EKS PVC issues] If MinIO doesn't appear at all, it's possible it was disabled in your
values.yaml
(
minio.enabled: false
). In that case, you'd need to ensure you have an alternative storage backend (like S3) configured under
global.storage
in your V2
values.yaml
. [chart V2 values] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.
p
@kapa.ai, i delted the logs. jobs are still stuck in starting status for the streams
k
Based on the knowledge sources, there is a documented case of jobs stuck after upgrading to Airbyte V2 (Helm chart) where deleting logs alone didn't resolve the issue. The fix reported by users in that case was related to a specific configuration problem after the V2 upgrade. [No jobs starting V2] Specifically, that issue references a workaround documented in this related GitHub issue (68115). Unfortunately, the knowledge sources don't include the full details of that workaround comment. A few things worth checking based on what the knowledge sources do cover: 1. Check if the MinIO pod is actually healthy now after your PVC fix:
Copy code
kubectl get pods -n airbyte
   kubectl describe pod airbyte-minio-0 -n airbyte
2. Check the
workload-api-server
replicas
— multiple replicas can cause race conditions in workload state updates that leave jobs stuck: [Jobs not removed K8s]
Copy code
kubectl get deployment -n airbyte | grep workload
3. Check the
dataplane_group_id
configuration — this is flagged as a common misconfiguration in V2 deployments that can cause jobs to not start. [Jobs not removed K8s] I'd recommend reviewing the GitHub issue #68115 directly for the specific V2 workaround, as the knowledge sources don't contain the full details of that fix. [No jobs starting V2] 👍 0 button 👎 0 button ℹ️ Tip: Mention @kapa.ai in the thread for follow-ups or DM the app for 1-to-1 help.