Hi everyone, I am trying to deploying pyflink appl...
# troubleshooting
r
Hi everyone, I am trying to deploying pyflink application in kubernetes. I am facing issue when trying to configure checkpointing. I have 2 task manager pods and 3.5 gb memory per task manager. This configuration works fine when I do not have checkpointing. When I configure checkpointing, I get OOMKilled in task manager pod and when I configure unaligned-checkpointing, I get sigsegv crash. This is my deployment file.
Copy code
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"
    taskmanager.numberOfTaskSlots: "2"

    # High Availability
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: file:///flink-data/ha

    # Checkpoints and Savepoints
    state.checkpoints.dir: file:///flink-data/checkpoints
    state.savepoints.dir: file:///flink-data/savepoints
    state.backend.type: rocksdb
    
    execution.checkpointing.incremental: "true"
   # execution.checkpointing.alignment-timeout: "3000"  # ms, after which checkpoint switches to unaligned mode dynamically

    #execution.checkpointing.unaligned: "true"                  # Enable unaligned checkpoints

    #rest.profiling.enabled: "true"
    #env.java.opts.taskmanager: "-Dorg.apache.beam.sdk.fn=DEBUG -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG"

  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "http://myminio-hl.flink.svc.cluster.local:9000"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
            - name: MAX_PARALLELISM
              value: "128"
          volumeMounts:
            - mountPath: /flink-data
              name: flink-pvc
      volumes:
        - name: flink-pvc
          persistentVolumeClaim:
            claimName: flink-data-pvc
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999

  jobManager:
    resource:
      memory: "3072m"
      cpu: 0.4

  taskManager:
    resource:
      memory: "3584m"
      cpu: 2
    replicas: 2

  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: stateless
I am not able find the root cause or the best way to configure checkpointing. Does anyone have any idea as to how I shall proceed with this?
p
Hello there! What's the expected size of your checkpoint? Keep in mind the checkpoint is stored in memory before actually happening. If your checkpoint size is too big, it may lead to OOM. Also, the frequency on the checkpointing interval may affect it. Something else perhaps worth looking at is incremental checkpoints.