Muazim Wani
05/20/2025, 6:19 PM{
"counts": {
"restored": 0,
"total": 0,
"in_progress": 0,
"completed": 0,
"failed": 0
},
"summary": {
"checkpointed_size": {
"min": 0,
"max": 0,
"avg": 0,
"p50": "NaN",
"p90": "NaN",
"p95": "NaN",
"p99": "NaN",
"p999": "NaN"
},
"state_size": {
"min": 0,
"max": 0,
"avg": 0,
"p50": "NaN",
"p90": "NaN",
"p95": "NaN",
"p99": "NaN",
"p999": "NaN"
},
"end_to_end_duration": {
"min": 0,
"max": 0,
"avg": 0,
"p50": "NaN",
"p90": "NaN",
"p95": "NaN",
"p99": "NaN",
"p999": "NaN"
},
"alignment_buffered": {
"min": 0,
"max": 0,
"avg": 0,
"p50": 0,
"p90": 0,
"p95": 0,
"p99": 0,
"p999": 0
},
"processed_data": {
"min": 0,
"max": 0,
"avg": 0,
"p50": "NaN",
"p90": "NaN",
"p95": "NaN",
"p99": "NaN",
"p999": "NaN"
},
"persisted_data": {
"min": 0,
"max": 0,
"avg": 0,
"p50": "NaN",
"p90": "NaN",
"p95": "NaN",
"p99": "NaN",
"p999": "NaN"
}
},
"latest": {
"completed": null,
"savepoint": null,
"failed": null,
"restored": null
},
"history": []
}
❌ Flink 1.20.1 (failing):
{
"errors": [
"org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n"
]
}
Also attaching the stack trace of Flink K8s operator
[m[33m2025-05-20 12:55:18,519[m [36mi.j.o.p.e.EventProcessor [m [1;31m[ERROR][perseus/test-spec-20250520125444-w2kvo] Error during event processing ExecutionScope{ resource id: ResourceID{name='test-spec-20250520125444-w2kvo', namespace='perseus'}, version: 15919105336}
org.apache.flink.kubernetes.operator.exception.ReconciliationException: Could not observe latest savepoint information
at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getLastCheckpoint(AbstractFlinkService.java:546)
at org.apache.flink.kubernetes.operator.observer.SnapshotObserver.observeLatestCheckpoint(SnapshotObserver.java:449)
at org.apache.flink.kubernetes.operator.observer.SnapshotObserver.observeSavepointStatus(SnapshotObserver.java:93)
at org.apache.flink.kubernetes.operator.observer.sessionjob.FlinkSessionJobObserver.observeInternal(FlinkSessionJobObserver.java:56)
at org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)
at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:111)
at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:57)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:153)
at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:111)
at org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:110)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:136)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:117)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:452)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)
at org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
]
at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getCheckpointInfo(AbstractFlinkService.java:581)
at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getLastCheckpoint(AbstractFlinkService.java:544)
... 18 more
Caused by: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)
at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)
at org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)
at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
]
at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:644)
at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$6(RestClient.java:628)
at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
... 3 more
❓ Question
Is this behavior change in Flink 1.20.1 expected for batch jobs in session clusters?
And if so, should the Flink K8s Operator handle this exception gracefully during reconciliation for batch-mode jobs, given that checkpointing is intentionally disabled?
Jobs in streaming mode and FlinkDeployment-based workloads work fine in both batch and streaming modes. This issue is limited to batch-mode jobs on session clusters.
Thanks in advance for any guidance!Muazim Wani
05/20/2025, 6:41 PMPedro Mázala
05/26/2025, 8:24 AMMuazim Wani
06/04/2025, 4:41 PM