Hi everyone, I am running a pyflink application in...
# troubleshooting
r
Hi everyone, I am running a pyflink application in Kubernetes. This is my config file.
Copy code
apiVersion: <http://flink.apache.org/v1beta1|flink.apache.org/v1beta1>
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: <http://624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest|624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest>
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"
    #taskmanager.numberOfTaskSlots: "2"

    # High Availability
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: file:///flink-data/ha

    # Checkpoints and Savepoints
    state.checkpoints.dir: file:///flink-data/checkpoints
    state.savepoints.dir: file:///flink-data/savepoints
    state.backend.type: rocksdb

    state.backend.rocksdb.memory.managed: "true"  # Enable Flink's memory management for RocksDB
    state.backend.rocksdb.memory.write-buffer-size: "64mb"  # Limit write buffer size
    state.backend.rocksdb.memory.max-file-size-to-compact: "2mb"  # Increase compaction file size
    state.backend.rocksdb.memory.min-files-to-compact: "3"  # Reduce compaction frequency
    state.backend.rocksdb.memory.max-files-to-compact: "10"
    state.backend.rocksdb.memory.max-output-file-size: "128mb"
    state.backend.rocksdb.memory.cache-capacity: "256mb"  # Reduce cache size
    state.backend.rocksdb.memory.managed: true
    state.backend.rocksdb.memory.write-buffer-ratio: "0.5"
    state.backend.rocksdb.memory.high-prio-pool-ratio: "0.1"


    python.fn-execution.bundle.size: "100"
    python.fn-execution.memory.managed: "true"
    python.fn-execution.buffer.memory.size: "64mb"

    #execution.checkpointing.alignment-timeout: "0"  # ms, after which checkpoint switches to unaligned mode dynamically

    #execution.checkpointing.unaligned: "true"                  # Enable unaligned checkpoints

    #rest.profiling.enabled: "true"
    #env.java.opts.taskmanager: "-Dorg.apache.beam.sdk.fn=DEBUG -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG"

  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: _PYTHON_WORKER_MEMORY_LIMIT
              value: "350000000"
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "<http://myminio-hl.flink.svc.cluster.local:9000>"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
            - name: MAX_PARALLELISM
              value: "128"
          volumeMounts:
            - mountPath: /flink-data
              name: flink-pvc
      volumes:
        - name: flink-pvc
          persistentVolumeClaim:
            claimName: flink-data-pvc
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999

  jobManager:
    resource:
      memory: "3072m"
      cpu: 3

  taskManager:
    resource:
      memory: "4096m"
      cpu: 1
    replicas: 32

  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: stateless
I have disabled checkpointing purposely. The issue is I have kept parallelism as 32 and have multiple processes. Now I am seeing in all the task manager logs that the process is completed simultaneously but in kafka partitions the data is received one after another with lot of delay. My output is only a single data point sinked to kafka which is link to a s3 bucket. Why am I not seeing all the data in kafka partitions simultaneously is data is processed simultaneously. (the parallelism is utilized by producing different keys to different partitions)
Copy code
KafkaSink.builder()
        .set_bootstrap_servers(BOOTSTRAP_SERVERS)
        .set_record_serializer(
            KafkaRecordSerializationSchema.builder()
            .set_topic(topic)
            .set_value_serialization_schema(
                JsonRowSerializationSchema.builder()
                .with_type_info(type_info)
                .build()
            )
            .build()
        )
        .set_delivery_guarantee(DeliveryGuarantee.AT_LEAST_ONCE)
        .build()
this is my kafka sink. I am using flink 1.20 Any help is greatly appreciated. Thankyou.
t
How many partitions does your sink kafka topic have? What is your job input, is that a Kafka topic too?
r
Partition are always kept equal to the parallelism. Input is a message on kafka topic which is source to the flink environment.