https://flink.apache.org/ logo
Join Slack
Powered by
hi team, can someone help me troubleshoot why my ...
# troubleshooting
c
hi team, can someone help me troubleshoot why my TM keeps saying i have duplicate configmap
Copy code
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: POST at: <https://10.100.0.1/api/v1/namespaces/trino-test/pods>. Message: Pod "clen-pipeline-taskmanager-1-13" is invalid: spec.volumes[2].name: Duplicate value: "hadoop-config-volume". Received status: Status(apiVersion=v1, code=422, details=StatusDetails(causes=[StatusCause(field=spec.volumes[2].name, message=Duplicate value: "hadoop-config-volume", reason=FieldValueDuplicate, additionalProperties={})], group=null, kind=Pod, name=clen-pipeline-taskmanager-1-13, retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status, message=Pod "clen-pipeline-taskmanager-1-13" is invalid: spec.volumes[2].name: Duplicate value: "hadoop-config-volume", metadata=ListMeta(_continue=null, remainingItemCount=null, resourceVersion=null, selfLink=null, additionalProperties={}), reason=Invalid, status=Failure, additionalProperties={}).
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:684) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:664) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:615) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:558) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleCreate(OperationSupport.java:308) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:644) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleCreate(BaseOperation.java:83) ~[flink-dist-1.16.2.jar:1.16.2]
	at io.fabric8.kubernetes.client.dsl.base.CreateOnlyResourceOperation.create(CreateOnlyResourceOperation.java:61) ~[flink-dist-1.16.2.jar:1.16.2]
	at org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$createTaskManagerPod$1(Fabric8FlinkKubeClient.java:163) ~[flink-dist-1.16.2.jar:1.16.2]
i m using kustomize so the final manifest would be
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: clen-pipeline
  namespace: trino-test
spec:
  flinkConfiguration:
    akka.ask.timeout: 60 min
    execution.checkpointing.interval: 5m
    execution.checkpointing.max-concurrent-checkpoints: "1"
    execution.checkpointing.min-pause: 5m
    execution.checkpointing.mode: EXACTLY_ONCE
    execution.checkpointing.snapshot-compression: "true"
    execution.checkpointing.timeout: 5m
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: s3://..
    jmx.server.port: "50103"
    state.checkpoints.dir: s3://..
    state.savepoints.dir: s3://..
    taskmanager.numberOfTaskSlots: "2"
  flinkVersion: v1_16
  image: clen/flink:hudi_and_flink_0.2
  job:
    jarURI: local:///opt/flink/examples/streaming/StateMachineExample.jar
    parallelism: 2
    state: running
    upgradeMode: stateless
  jobManager:
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: job-manager-pod-template
      spec:
        containers:
        - livenessProbe:
            initialDelaySeconds: 30
            periodSeconds: 60
            tcpSocket:
              port: 6123
          readinessProbe:
            initialDelaySeconds: 15
            periodSeconds: 15
            tcpSocket:
              port: 6123
    replicas: 2
    resource:
      cpu: 0.5
      memory: 1024m
  podTemplate:
    metadata:
      annotations:
        <http://sidecar.istio.io/inject|sidecar.istio.io/inject>: "true"
        <http://traffic.sidecar.istio.io/excludeInboundPorts|traffic.sidecar.istio.io/excludeInboundPorts>: 2552, 8558, 6123
        <http://traffic.sidecar.istio.io/excludeOutboundPorts|traffic.sidecar.istio.io/excludeOutboundPorts>: 2552, 8558, 6123
    spec:
      containers:
      - name: flink-main-container
        volumeMounts:
        - mountPath: /opt/hadoop/etc/hadoop/core-site.xml
          name: hadoop-config-volume
          subPath: core-site.xml
      volumes:
      - configMap:
          name: corexml
        name: hadoop-config-volume
  serviceAccount: flink
  taskManager:
    podTemplate:
      apiVersion: v1
      kind: Pod
      metadata:
        name: task-manager-pod-template
      spec:
        containers:
        - livenessProbe:
            initialDelaySeconds: 30
            periodSeconds: 60
            tcpSocket:
              port: 6122
          readinessProbe:
            initialDelaySeconds: 15
            periodSeconds: 15
            tcpSocket:
              port: 6122
    replicas: 2
    resource:
      cpu: 2
      memory: 4096m
Copy code
clen.moras@MBT727VPGY21 dev % kubectl describe configmap corexml
Name:         corexml
Namespace:    trino-test
Labels:       <none>
Annotations:  <none>

Data
====
core-site.xml:
----
<configuration>
    <property>
        <name>fs.s3a.impl</name>
        <value>org.apache.hadoop.fs.s3a.S3AFileSystem</value>
    </property>
    <!-- Optionally, set other S3 configurations, e.g., region, endpoint, etc. -->
    <property>
        <name>fs.s3a.region</name>
        <value>us-east-1</value>
    </property>
    <!-- Set the buffer directory for S3A -->
    <property>
        <name>fs.s3a.buffer.dir</name>
        <value>/tmp</value>
    </property>
    <property>
        <name>fs.s3a.aws.credentials.provider</name>
        <value>com.amazonaws.auth.WebIdentityTokenCredentialsProvider</value>
    </property>
</configuration>


BinaryData
====

Events:  <none>
4 Views
Open in Slack
PreviousNext
Powered by