Subject: Flink 1.20.1 + Flink K8s Operator 1.10: R...
# troubleshooting
m
Subject: Flink 1.20.1 + Flink K8s Operator 1.10: Reconciliation Failure in Batch Mode for Session Jobs Hi Everyone, We've recently upgraded our Flink environment as follows: Flink session clusters: from 1.16.3 → 1.20.1 Flink Kubernetes Operator: from 1.5 → 1.10 After this upgrade, we’ve observed a discrepancy in behavior between the older and newer Flink session clusters when running jobs in ExecutionMode = BATCH. We currently run two session clusters. On both clusters, the jobs complete successfully in batch mode — they are marked as FINISHED in the Flink UI, and all processing is completed. However, during the reconciliation step in the Flink K8s Operator: For the 1.16.3 cluster, the operator reconciles the job successfully. For the 1.20.1 cluster, the operator fails with the error: "Could not observe latest savepoint information". Since these are batch mode jobs with stateless upgrade mode, we do not expect savepoints/checkpoints to exist, and indeed, no checkpointing is configured. Upon investigation, it appears that in Flink k8s operator (LINK) when making the call to the session cluster to get the checkpoints info , below behavior was observed: In Flink 1.16.3, the /jobs/:jobId/checkpoints endpoint returns a valid response with dummy/default values. In Flink 1.20.1, the same endpoint throws a RestHandlerException: Checkpointing has not been enabled. Here are the API responses for the same batch-mode job on both versions: ✔️ Flink 1.16.3 (working):
Copy code
{
  "counts": {
    "restored": 0,
    "total": 0,
    "in_progress": 0,
    "completed": 0,
    "failed": 0
  },
  "summary": {
    "checkpointed_size": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": "NaN",
      "p90": "NaN",
      "p95": "NaN",
      "p99": "NaN",
      "p999": "NaN"
    },
    "state_size": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": "NaN",
      "p90": "NaN",
      "p95": "NaN",
      "p99": "NaN",
      "p999": "NaN"
    },
    "end_to_end_duration": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": "NaN",
      "p90": "NaN",
      "p95": "NaN",
      "p99": "NaN",
      "p999": "NaN"
    },
    "alignment_buffered": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": 0,
      "p90": 0,
      "p95": 0,
      "p99": 0,
      "p999": 0
    },
    "processed_data": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": "NaN",
      "p90": "NaN",
      "p95": "NaN",
      "p99": "NaN",
      "p999": "NaN"
    },
    "persisted_data": {
      "min": 0,
      "max": 0,
      "avg": 0,
      "p50": "NaN",
      "p90": "NaN",
      "p95": "NaN",
      "p99": "NaN",
      "p999": "NaN"
    }
  },
  "latest": {
    "completed": null,
    "savepoint": null,
    "failed": null,
    "restored": null
  },
  "history": []
}
❌ Flink 1.20.1 (failing):
Copy code
{
  "errors": [
    "org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)\n\tat org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)\n\tat java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)\n\tat java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)\n\tat java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)\n\tat java.base/java.util.concurrent.FutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\n"
  ]
}
Also attaching the stack trace of Flink K8s operator
Copy code
[m[33m2025-05-20 12:55:18,519[m [36mi.j.o.p.e.EventProcessor      [m [1;31m[ERROR][perseus/test-spec-20250520125444-w2kvo] Error during event processing ExecutionScope{ resource id: ResourceID{name='test-spec-20250520125444-w2kvo', namespace='perseus'}, version: 15919105336}
org.apache.flink.kubernetes.operator.exception.ReconciliationException: Could not observe latest savepoint information
	at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getLastCheckpoint(AbstractFlinkService.java:546)
	at org.apache.flink.kubernetes.operator.observer.SnapshotObserver.observeLatestCheckpoint(SnapshotObserver.java:449)
	at org.apache.flink.kubernetes.operator.observer.SnapshotObserver.observeSavepointStatus(SnapshotObserver.java:93)
	at org.apache.flink.kubernetes.operator.observer.sessionjob.FlinkSessionJobObserver.observeInternal(FlinkSessionJobObserver.java:56)
	at org.apache.flink.kubernetes.operator.observer.AbstractFlinkResourceObserver.observe(AbstractFlinkResourceObserver.java:49)
	at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:111)
	at org.apache.flink.kubernetes.operator.controller.FlinkSessionJobController.reconcile(FlinkSessionJobController.java:57)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:153)
	at io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:111)
	at org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)
	at io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:110)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:136)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:117)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:91)
	at io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:64)
	at io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:452)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
Caused by: java.util.concurrent.ExecutionException: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
]
	at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
	at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getCheckpointInfo(AbstractFlinkService.java:581)
	at org.apache.flink.kubernetes.operator.service.AbstractFlinkService.getLastCheckpoint(AbstractFlinkService.java:544)
	... 18 more
Caused by: org.apache.flink.runtime.rest.util.RestClientException: [org.apache.flink.runtime.rest.handler.RestHandlerException: Checkpointing has not been enabled.
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.createCheckpointingStatistics(CheckpointingStatisticsHandler.java:107)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:85)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.CheckpointingStatisticsHandler.handleCheckpointStatsRequest(CheckpointingStatisticsHandler.java:59)
	at org.apache.flink.runtime.rest.handler.job.checkpoints.AbstractCheckpointStatsHandler.lambda$handleRequest$1(AbstractCheckpointStatsHandler.java:89)
	at java.base/java.util.concurrent.CompletableFuture$UniApply.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Unknown Source)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at java.base/java.lang.Thread.run(Unknown Source)
]
	at org.apache.flink.runtime.rest.RestClient.parseResponse(RestClient.java:644)
	at org.apache.flink.runtime.rest.RestClient.lambda$submitRequest$6(RestClient.java:628)
	at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
	at java.base/java.util.concurrent.CompletableFuture$Completion.run(Unknown Source)
	... 3 more
❓ Question Is this behavior change in Flink 1.20.1 expected for batch jobs in session clusters? And if so, should the Flink K8s Operator handle this exception gracefully during reconciliation for batch-mode jobs, given that checkpointing is intentionally disabled? Jobs in streaming mode and FlinkDeployment-based workloads work fine in both batch and streaming modes. This issue is limited to batch-mode jobs on session clusters. Thanks in advance for any guidance!
This seems to be fixed in newer version of flink k8s operator https://github.com/apache/flink-kubernetes-operator/pull/948
p
Have you tested the solution?
m
Yes, @Pedro Mázala. The jobs with the Execution mode Batch are now reconciled as expected. A new Flink operator version is also released, 1.12.0, which also contains the same fix https://lists.apache.org/thread/47dg33kw6gyjdz7j90l5v7qy80d1p5pz