Hi everyone, I am trying to deploy Flink using Kub...
# troubleshooting
r
Hi everyone, I am trying to deploy Flink using Kubernetes operator. After introducing checkpointing my task manager manager pods are failing continuously on same resource and same work load. I tried using profiling to find the root cause but I could only see spike in resource consumption once the before the task manager pods fail.
Copy code
kind: FlinkDeployment
metadata:
  name: data-processing-flink
  namespace: flink
spec:
  image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
  flinkVersion: "v1_20"
  serviceAccount: flink
  flinkConfiguration:
    rest.port: "8081"
    jobmanager.rpc.address: "data-processing-flink"
    jobmanager.rpc.port: "6123"
    taskmanager.numberOfTaskSlots: "2"

    # High Availability
    high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
    high-availability.storageDir: file:///flink-data/ha

    # Checkpoints and Savepoints
    state.checkpoints.dir: file:///flink-data/checkpoints
    state.savepoints.dir: file:///flink-data/savepoints
    state.backend: rocksdb
    state.backend.incremental: true

    rest.profiling.enabled: "true"
    env.java.opts.taskmanager: "-Dorg.apache.beam.sdk.fn=DEBUG -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG"

  podTemplate:
    spec:
      imagePullSecrets:
        - name: ecrscr-credentials
      containers:
        - name: flink-main-container
          imagePullPolicy: Always
          envFrom:
            - secretRef:
                name: redis-secret-main
          env:
            - name: KAFKA_BROKER
              value: "kafka.flink.svc.cluster.local:9092"
            - name: BOOTSTRAP_SERVERS
              value: "kafka.flink.svc.cluster.local:9092"
            - name: MINIO_SERVER
              value: "http://myminio-hl.flink.svc.cluster.local:9000"
            - name: MINIO_USER
              value: "minio"
            - name: MINIO_PASSWORD
              value: "minio123"
            - name: MAX_PARALLELISM
              value: "128"
          volumeMounts:
            - mountPath: /flink-data
              name: flink-pvc
      volumes:
        - name: flink-pvc
          persistentVolumeClaim:
            claimName: flink-data-pvc
      securityContext:
        runAsUser: 9999
        runAsGroup: 9999

  jobManager:
    resource:
      memory: "3072m"
      cpu: 0.4

  taskManager:
    resource:
      memory: "3584m"
      cpu: 2
    replicas: 2

  job:
    entryClass: "org.apache.flink.client.python.PythonDriver"
    args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
    upgradeMode: last-state
This is the deployment file I am using. My application is already consuming a lot of resources and I cannot afford allocating more resources to it. Also the checkpointing size is also 2-3 mb for a few successful checkpoint that I observed. Can anyone guide from where can I start to debug this issue. Thanks
a
I terms of troubleshooting you should look at the logs of the task and job manager. As for a potential cause you can consider what the overall end-to-end latency for records are. If the end-to-end record latency exceeds the
checkpointing.timeout
(default is 10min) then the checkpoint will timeout and the job will be cancelled (remember that checkpoint barriers cannot overtake the records). You can increase the timeout using
execution.checkpointing.timeout
and allow the job to continue even when checkpoints do fail using
execution.checkpointing.tolerable-failed-checkpoints
(https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/)
If the cause indeed is checkpointing timeouts, then another option to consider is using unaligned checkpoints, "_which greatly reduce checkpointing times under backpressure._" (again, see https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/)
r
@Andreas Bube Thanks for the reply. I have checked for the logs
a
@Rushikesh Gulve If you are able to provide the job and taskmanager logs from when it last failed, I'd be willing to give them a quick glance to see if I can spot any obvious clues.
r
@Andreas Bube These are my latest Task Manager logs. I have configured 2 Task Managers. Logs are for each Task Manager. Unfortunately I dont have Job Manager logs. If you could suggest anything, it would be great help.
a
@Rushikesh Gulve You don't happen to have logs that capture the issue? From these logs all I see a job that is running. I don't see any failures?
r
@Andreas Bube I took these logs 2 days back. These logs have some different logging. Can glance through this once if it is of any help? Apologies for the trouble.
a
This taskmanager looks to have been shut down with a remoting call:
Copy code
RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
I'm gusing the other taskmanager or jobmanager had an issue. If you have the corresponding logs from those it would be helpful. It could also perhaps be k8s, Karpenter etc, etc, that could have decided to kill the pod for whatever reason.
r
Unfortunately, I don't have those logs. I will try to reproduce the error and get the logs. Thankyou for your hep
πŸ‘ 1
a
Sure. Ping me and I'll have a look when my schedule permits
r
@Andreas Bube I have got the logs of task manager. I could see the issue of JVM and RocksDB but I am not able to find the root cause. I could not find any hint from github issues as well. If you have any idea of issue, Can you take a look at it?
a
Thanks for the log. Yeah. I can tell what breaks. It's issue with some native code is all I'm getting from the logs.
Copy code
# JRE version: OpenJDK Runtime Environment Temurin-11.0.27+6 (11.0.27+6) (build 11.0.27+6)
#  SIGSEGV (0xb) at pc=0x00007f8da33ff898, pid=1, tid=2234
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.27+6 (11.0.27+6, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libc.so.6+0x28898]  abort+0x178
The logs also states a error report is created at
/opt/flink/usrlib/hs_err_pid1.log
. Maybe that has clues as to which component is at fault for the illegal memory access (the SIGSEGV) I don't think I can help much further than this. I imagine some component of your setup has a bug. Whether it's Flink, a plugin, Beam, JRE or your flink application code I cannot tell.
r
Thanks @Andreas Bube I got the issue. It was due to unaligned checkpoints which produce many threads and leading to SIGSEGV as you mentioned. It needs some configuration changes to handle resources. Thanks a lot for your help.
g
would be keen if you be willing t share your entire Flink stack yaml again... have a k8s cluster also and would like to deploy Flink on it. Makes for some interesting lab'ing.