Hi team! I'm trying to mount a volume to `flink-ma...
# troubleshooting
s
Hi team! I'm trying to mount a volume to
flink-main-container
, the volume is defined by some annotation injection. Here is the podTemplate snippet, but it gives volumeMount name not found:
Copy code
podTemplate:
   ....   
     containers:
        - name: flink-main-container
          volumeMounts:
            - mountPath: /opt/flink/log
              name: flink-logs
            - mountPath: /opt/conf
              name: isc-confbundle
      volumes:
        - name: flink-logs
          emptyDir: { }
--------
Error: {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"org.apache.flink.client.deployment.ClusterDeploymentException: Could not create Kubernetes cluster \"event-router\".","throwableList":[{"type":"org.apache.flink.client.deployment.ClusterDeploymentException","message":"Could not create Kubernetes cluster \"event-router\"."},{"type":"org.apache.flink.kubernetes.shaded.io.fabric8.kubernetes.client.KubernetesClientException","message":"Failure executing: POST at: <https://192.168.0.1/apis/apps/v1/namespaces/data-infra-event-router/deployments>. Message: Deployment.apps \"event-router\" is invalid: spec.template.spec.containers[0].volumeMounts[1].name: Not found: \"isc-confbundle\". Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.template.spec.containers[0].volumeMounts[1].name, message=Not found: \"isc-confbundle\", reason=FieldValueNotFound, additionalProperties={})], group=apps, kind=Deployment, name=event-router, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Deployment.apps \"event-router\" is invalid: spec.template.spec.containers[0].volumeMounts[1].name: Not found: \"isc-confbundle\", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={})."}]}
So I tried manually add this volumes as below, then the error turns out to be duplicate volume name(so I guess the volume should be there?)
Copy code
podTemplate:
   ....   
     containers:
        - name: flink-main-container
          volumeMounts:
            - mountPath: /opt/flink/log
              name: flink-logs
            - mountPath: /opt/conf
              name: isc-confbundle
      volumes:
        - name: flink-logs
          emptyDir: { }
        - name: isc-confbundle
          emptyDir: { }
--------
Error:  
[ERROR][data-infra-event-router/event-router] Flink Deployment failed
org.apache.flink.kubernetes.operator.exception.DeploymentFailedException: Pod "event-router-66b68d8b7b-sq6qr" is invalid: spec.volumes[6].name: Duplicate value: "isc-confbundle"
Any advice here?
1
I guess this might be related to the execution order of Flink operator submit the pod deployment and the annotation injection?
we figured this out, this is a issue with k8s deployment, we changed our annotation injection logic, and it works
c
interesting. I have a similar issue, where in the cluster says it duplicate
Copy code
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.100.0.1/api/v1/namespaces/trino-test/pods>. Message: Pod "clen-pipeline-taskmanager-1-13" is invalid: spec.volumes[2].name: Duplicate value: "hadoop-config-volume". Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.volumes[2].name, message=Duplicate value: "hadoop-config-volume", reason=FieldValueDuplicate, additionalProperties={})], group=null, kind=Pod, name=clen-pipeline-taskmanager-1-13, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "clen-pipeline-taskmanager-1-13" is invalid: spec.volumes[2].name: Duplicate value: "hadoop-config-volume", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:615) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[flink-dist-1.16.2.jar:1.16.2]
	at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163) ~[flink-dist-1.16.2.jar:1.16.2]
Copy code
podTemplate:
    metadata:
      annotations:
        <http://sidecar.istio.io/inject|sidecar.istio.io/inject>: "true"
        <http://traffic.sidecar.istio.io/excludeInboundPorts|traffic.sidecar.istio.io/excludeInboundPorts>: 2552, 8558, 6123
        <http://traffic.sidecar.istio.io/excludeOutboundPorts|traffic.sidecar.istio.io/excludeOutboundPorts>: 2552, 8558, 6123
    spec:
      containers:
      - name: flink-main-container
        volumeMounts:
        - mountPath: /opt/hadoop/etc/hadoop/core-site.xml
          name: clen-config
          subPath: core-site.xml
      volumes:
      - configMap:
          name: corexml
        name: clen-config