Hello, I am running Flink jobs (Flink version 1.16...
# troubleshooting
s
Hello, I am running Flink jobs (Flink version 1.16) and running on Kubernetes cluster using flink-kubernetes-operator-1.8.0. The upgrade of the flink job is
savepoint
. Following savepoint related properties are configured as well
Copy code
kubernetes.operator.job.drain-on-savepoint-deletion: "true"
    kubernetes.operator.job.savepoint-on-deletion: "true"
The issue is we are observing that certain jobs,
stop-with-savepoint
operation fails with error
Copy code
2025-03-28 15:04:22,142 INFO  org.apache.flink.runtime.jobmaster.JobMaster                 [] - Triggering stop-with-savepoint for job 5f23048f697a3487dc32641bc37f129f.
2025-03-28 15:04:22,155 INFO  org.apache.flink.runtime.checkpoint.CheckpointCoordinator    [] - Triggering checkpoint 17 (type=SavepointType{name='Terminate Savepoint', postCheckpointAction=TERMINATE, formatType=CANONICAL}) @ 1743174262143 for job 5f23048f697a3487dc32641bc37f129f.
2025-03-28 15:04:23,207 INFO  org.apache.flink.runtime.executiongraph.ExecutionGraph       [] - GlobalWindowAggregate[14] -> Calc[15] -> ConstraintEnforcer[16] -> *anonymous_kafka$2*[16]: Writer -> *anonymous_kafka$2*[16]: Committer (8/8) (d444a51a4e7b2d8b2bf5395bf97d3f36_f6dc7f4d2283f4605b127b9364e21148_7_0) switched from RUNNING to FAILED on sg-demo-taskmanager-1-1 @ ip-100-64-92-59.ec2.internal (dataPort=43769).
java.lang.UnsupportedOperationException: Currently it is not supported to update the CommittableSummary for a checkpoint coming from the same subtask. Please check the status of FLINK-25920
While we understand this issue is addressed in Flink 1.19 and above, the problem is sometimes the job get stuck when we try to change FlinkDeployment state to SUSPENDED. On examining the logs, we observed that flink job repeatedly try to capture a savepoint but fails to do so due to above error. Overall Flink job keeps running in the background. See attached image for checkpoint and savepoint trail. Is there any configuration which can be used to terminate the after 1 or 2 attempts to capture savepoint? I tried to reproduce this issue with same stack but the observation is that on stopping the job, savepoint does fail due to aforementioned issue, but the job does terminate.
a
Hi soumya, im facing a similar issue on flink 1.17 and flink kubernetes operator 1.6, happens with some of the jobs not all. when we try to restart the job with savepoint, the ongoing checkpoints start failing and eventually the job gets stuck and i have to manually delete the deployment to restart the job. these are the only errors i get on the job
Copy code
at java.lang.Thread.run(Unknown Source) [?:?]
    at org.apache.flink.runtime.taskmanager.Task.run(Task.java:562) [flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:745) [flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:931) [flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:952) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:788) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:839) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:229) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:352) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMailsWhenDefaultActionUnavailable(MailboxProcessor.java:367) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMail(MailboxProcessor.java:398) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:90) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:50) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointOperation$19(StreamTask.java:1410) ~[flink-dist-1.17.2.jar:1.17.2]
    at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$notifyCheckpointAbortAsync$17(StreamTask.java:1385) ~[flink-dist-1.17.2.jar:1.17.2]
org.apache.flink.util.FlinkRuntimeException: Stop-with-savepoint failed.
some taskmanagers restart and some are still stuck and the consumption drops to 0