Soumya Ghosh
03/28/2025, 3:28 PMsavepoint .
Following savepoint related properties are configured as well
kubernetes.operator.job.drain-on-savepoint-deletion: "true"
kubernetes.operator.job.savepoint-on-deletion: "true"
The issue is we are observing that certain jobs, stop-with-savepoint operation fails with error
2025-03-28 15:04:22,142 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Triggering stop-with-savepoint for job 5f23048f697a3487dc32641bc37f129f.
2025-03-28 15:04:22,155 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - Triggering checkpoint 17 (type=SavepointType{name='Terminate Savepoint', postCheckpointAction=TERMINATE, formatType=CANONICAL}) @ 1743174262143 for job 5f23048f697a3487dc32641bc37f129f.
2025-03-28 15:04:23,207 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - GlobalWindowAggregate[14] -> Calc[15] -> ConstraintEnforcer[16] -> *anonymous_kafka$2*[16]: Writer -> *anonymous_kafka$2*[16]: Committer (8/8) (d444a51a4e7b2d8b2bf5395bf97d3f36_f6dc7f4d2283f4605b127b9364e21148_7_0) switched from RUNNING to FAILED on sg-demo-taskmanager-1-1 @ ip-100-64-92-59.ec2.internal (dataPort=43769).
java.lang.UnsupportedOperationException: Currently it is not supported to update the CommittableSummary for a checkpoint coming from the same subtask. Please check the status of FLINK-25920
While we understand this issue is addressed in Flink 1.19 and above, the problem is sometimes the job get stuck when we try to change FlinkDeployment state to SUSPENDED.
On examining the logs, we observed that flink job repeatedly try to capture a savepoint but fails to do so due to above error. Overall Flink job keeps running in the background.
See attached image for checkpoint and savepoint trail.
Is there any configuration which can be used to terminate the after 1 or 2 attempts to capture savepoint?
I tried to reproduce this issue with same stack but the observation is that on stopping the job, savepoint does fail due to aforementioned issue, but the job does terminate.Anmol
05/22/2025, 8:02 AMat java.lang.Thread.run(Unknown Source) [?:?]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) [flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) [flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$19(StreamTask.java:1410) ~[flink-dist-1.17.2.jar:1.17.2]
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointAbortAsync$17(StreamTask.java:1385) ~[flink-dist-1.17.2.jar:1.17.2]
org.apache.flink.util.FlinkRuntimeException: Stop-with-savepoint failed.Anmol
05/22/2025, 8:03 AM