Rushikesh Gulve
06/16/2025, 6:47 AMkind: FlinkDeployment
metadata:
name: data-processing-flink
namespace: flink
spec:
image: 624083781302.dkr.ecr.ap-south-1.amazonaws.com/product/data-streaming-flink:latest
flinkVersion: "v1_20"
serviceAccount: flink
flinkConfiguration:
rest.port: "8081"
jobmanager.rpc.address: "data-processing-flink"
jobmanager.rpc.port: "6123"
taskmanager.numberOfTaskSlots: "2"
# High Availability
high-availability: org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
high-availability.storageDir: file:///flink-data/ha
# Checkpoints and Savepoints
state.checkpoints.dir: file:///flink-data/checkpoints
state.savepoints.dir: file:///flink-data/savepoints
state.backend: rocksdb
state.backend.incremental: true
rest.profiling.enabled: "true"
env.java.opts.taskmanager: "-Dorg.apache.beam.sdk.fn=DEBUG -Dorg.apache.beam.runners.fnexecution.control.FnApiControlClient=DEBUG"
podTemplate:
spec:
imagePullSecrets:
- name: ecrscr-credentials
containers:
- name: flink-main-container
imagePullPolicy: Always
envFrom:
- secretRef:
name: redis-secret-main
env:
- name: KAFKA_BROKER
value: "kafka.flink.svc.cluster.local:9092"
- name: BOOTSTRAP_SERVERS
value: "kafka.flink.svc.cluster.local:9092"
- name: MINIO_SERVER
value: "http://myminio-hl.flink.svc.cluster.local:9000"
- name: MINIO_USER
value: "minio"
- name: MINIO_PASSWORD
value: "minio123"
- name: MAX_PARALLELISM
value: "128"
volumeMounts:
- mountPath: /flink-data
name: flink-pvc
volumes:
- name: flink-pvc
persistentVolumeClaim:
claimName: flink-data-pvc
securityContext:
runAsUser: 9999
runAsGroup: 9999
jobManager:
resource:
memory: "3072m"
cpu: 0.4
taskManager:
resource:
memory: "3584m"
cpu: 2
replicas: 2
job:
entryClass: "org.apache.flink.client.python.PythonDriver"
args: ["-pyclientexec", "/usr/bin/python3", "-py", "/opt/flink/usrlib/data_processing_v9/stream_processing/process_dependency/create_env.py"]
upgradeMode: last-state
This is the deployment file I am using. My application is already consuming a lot of resources and I cannot afford allocating more resources to it. Also the checkpointing size is also 2-3 mb for a few successful checkpoint that I observed. Can anyone guide from where can I start to debug this issue. ThanksAndreas Bube
06/16/2025, 7:03 AMcheckpointing.timeout (default is 10min) then the checkpoint will timeout and the job will be cancelled (remember that checkpoint barriers cannot overtake the records). You can increase the timeout using execution.checkpointing.timeout and allow the job to continue even when checkpoints do fail using execution.checkpointing.tolerable-failed-checkpoints (https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/fault-tolerance/checkpointing/)Andreas Bube
06/16/2025, 7:18 AMRushikesh Gulve
06/16/2025, 8:03 AMAndreas Bube
06/16/2025, 8:40 AMRushikesh Gulve
06/16/2025, 8:50 AMAndreas Bube
06/16/2025, 8:54 AMRushikesh Gulve
06/16/2025, 8:58 AMAndreas Bube
06/16/2025, 9:03 AMRECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
I'm gusing the other taskmanager or jobmanager had an issue. If you have the corresponding logs from those it would be helpful.
It could also perhaps be k8s, Karpenter etc, etc, that could have decided to kill the pod for whatever reason.Rushikesh Gulve
06/16/2025, 9:07 AMAndreas Bube
06/16/2025, 9:08 AMRushikesh Gulve
06/17/2025, 11:05 AMAndreas Bube
06/17/2025, 12:14 PM# JRE version: OpenJDK Runtime Environment Temurin-11.0.27+6 (11.0.27+6) (build 11.0.27+6)
# SIGSEGV (0xb) at pc=0x00007f8da33ff898, pid=1, tid=2234
# Java VM: OpenJDK 64-Bit Server VM Temurin-11.0.27+6 (11.0.27+6, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libc.so.6+0x28898] abort+0x178
The logs also states a error report is created at /opt/flink/usrlib/hs_err_pid1.log. Maybe that has clues as to which component is at fault for the illegal memory access (the SIGSEGV)
I don't think I can help much further than this. I imagine some component of your setup has a bug. Whether it's Flink, a plugin, Beam, JRE or your flink application code I cannot tell.Rushikesh Gulve
06/17/2025, 1:16 PMGeorge Leonard
06/30/2025, 5:46 AM