Hi everyone, I have a flink application deployed u...
# troubleshooting
r
Hi everyone, I have a flink application deployed using kubernetes operator. I have 40 task manager pods and I am using unaligned checkpointing. The tolerable failed checkpoint is 3. I got these logs in one of my task manager where task manager did not log anything for 15 minutes and then logged a SIGSEGV error. During this interval 3 checkpoints have failed and all task managers are killed.
Copy code
# JRE version: OpenJDK Runtime Environment Temurin-11.0.27+6 (11.0.27+6) (build 11.0.27+6)
2025-06-23 11:43:03.749	
#
2025-06-23 11:43:03.749	
#  SIGSEGV (0xb) at pc=0x00007fa5f75ff898, pid=1, tid=3479
2025-06-23 11:43:03.748	
#
2025-06-23 11:43:03.748	
# A fatal error has been detected by the Java Runtime Environment:
2025-06-23 11:43:03.748	
#
2025-06-23 11:43:03.748	
pthread lock: Invalid argument
2025-06-23 11:29:59.808	
2025-06-23 09:29:59,808 INFO  /usr/local/lib/python3.10/dist-packages/pyflink/fn_execution/beam/beam_sdk_worker_main.py:35 [] - after minio data_output: WindowProcessComputeIOStates-5-1736416800000, 64558e0e8caa4ebabc262d76e8117256, 1736416800000
During this time, other task managers are logging a failiure in checkpointing.
Copy code
org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestExecutorImpl.loop(ChannelStateWriteRequestExecutorImpl.java:182) ~[flink-dist-1.20.1.jar:1.20.1]
2025-06-23 11:33:00.661	
	at org.apache.flink.runtime.checkpoint.channel.ChannelStateWriteRequestDispatcherImpl.dispatchInternal(ChannelStateWriteRequestDispatcherImpl.java:127) ~[flink-dist-1.20.1.jar:1.20.1]
2025-06-23 11:33:00.661	
2025-06-23 09:33:00,634 INFO  org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - KEYED PROCESS -> (Extract-Timestamp -> Timestamps/Watermarks -> Remove-Timestamp -> Map -> Sink: Writer -> Sink: Committer, Map, Map -> Sink: Writer -> Sink: Committer) (5/40)#0 - asynchronous part of checkpoint 15 could not be completed.
2025-06-23 11:33:00.661	
2025-06-23 09:33:00,629 INFO  org.apache.flink.streaming.runtime.tasks.AsyncCheckpointRunnable [] - Source: prepareData->ProcessComputePartTypeTransactions -> _stream_key_by_map_operator (5/40)#0 - asynchronous part of checkpoint 15 could not be completed.
2025-06-23 11:33:00.661	
java.util.concurrent.ExecutionException: org.apache.flink.runtime.checkpoint.CheckpointException: The checkpoint was aborted due to exception of other subtasks sharing the ChannelState file.
I am guessing the one task manager whcih got unresponsive was responsible for the checkpoint failure which happened due to timeout. Can anyone help with identifying the root cause?