Hi All, I am trying to deploy my pyflink applicati...
# troubleshooting
r
Hi All, I am trying to deploy my pyflink application using Kubernetes. My process works fine for 1 Task Manager (2 core cpu and 4gb memory). Now I increase the Task Manager to 10 with same configuration for each Task Manager. Initially it works fine, but as soon as I increase TMs the job fails with error. Error in Job Manager logs are too long. I think the root cause is
Copy code
2025-06-04 07:28:49,048 WARN  org.apache.pekko.remote.ReliableDeliverySupervisor           [] - Association with remote system [pekko.tcp://flink@192.168.164.24:6122] has failed, address is now gated for [50] ms. Reason: [Association failed with [pekko.tcp://flink@192.168.164.24:6122]] Caused by: [org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AnnotatedConnectException: Connection refused: /192.168.164.24:6122, caused by: java.net.ConnectException: Connection refused]
I am not able to understand the cause of the issue. I am also sharing my deployment file
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"
    taskmanager.numberOfTaskSlots: "2"
  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "http://myminio-hl.flink.svc.cluster.local:9000"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999
  jobManager:
    resource:
      memory: "2048m"
      cpu: 1
  taskManager:
    resource:
      memory: "4096m"
      cpu: 2
    replicas: 10
  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: stateless
Can anyone suggest possible causes. Any help would be greatly appreciated.
p
This exception is thrown right before a disconnection between TM and JM. I think this may not be your root cause exception