the job manager was shut down via pod eviction: ``...
# troubleshooting
c
the job manager was shut down via pod eviction:
Copy code
2024-07-11 08:18:45,866 INFO  org.apache.flink.runtime.jobmaster.slotpool.DefaultDeclarativeSlotPool [] - Releasing slot [3c92e4ee1303da214e868fcef5d4ad91].
2024-07-11 08:19:08,335 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - RECEIVED SIGNAL 15: SIGTERM. Shutting down as requested.
2024-07-11 08:19:08,336 INFO  org.apache.flink.runtime.entrypoint.ClusterEntrypoint        [] - Shutting StandaloneApplicationClusterEntryPoint down with application status UNKNOWN. Diagnostics Cluster entrypoint has been closed externally..
d
Do you have
Copy code
restoreStrategy:
    kind: LATEST_STATE
setting? Are you using LATEST_STATE?
c
I am not aware of that setting. We are using
standalone Flink on Kubernetes
I cannot find restoreStrategy here
d
Are you using the Kubernetes Flink Operator for deployment?
c
no. I am using standalone FLink on Kubernetes, not native kubernetesn or flink kubernetes oeprator
this one
in application mode
d
Ok, I see so you have set this execution.checkpointing.externalized-checkpoint-retention policy
c
yes
So I assume when the cluster shuts down gracefully then the checkpoint and the configmap pointing to the checkpoint should remain intact so on the next startup the app can resume from it
d
And what is your setting for state.checkpoints.num-retained?
c
should be at the default of
1
d
Maybe that needs to be increased …
execution.checkpointing.interval ?
c
Copy code
execution.checkpointing.interval: 5 min
    execution.checkpointing.timeout: 5 min
    execution.checkpointing.unaligned.enabled: true
    execution.checkpointing.aligned-checkpoint-timeout: 5 min
    execution.checkpointing.tolerable-failed-checkpoints: 2
    execution.checkpointing.mode: EXACTLY_ONCE
    execution.checkpointing.max-concurrent-checkpoints: 1
    execution.checkpointing.min-pause: 5 min
    execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
this is the log message that I wonder about:
Copy code
2024-07-11 08:19:08,922 INFO  org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter [] - Shutting down.
2024-07-11 08:19:08,922 INFO  org.apache.flink.kubernetes.highavailability.KubernetesCheckpointIDCounter [] - Removing counter from ConfigMap flink-fuzzy-wifibissid-counter-pax-cluster-da512a1587e13e9ef844938ede5adab8-config-map
Copy code
/**
     * Checks whether this state is <i>globally terminal</i>. A globally terminal job is complete
     * and cannot fail any more and will not be restarted or recovered by another standby master
     * node.
     *
     * <p>When a globally terminal state has been reached, all recovery data for the job is dropped
     * from the high-availability services.
     *
     * @return True, if this job status is globally terminal, false otherwise.
     */
    public boolean isGloballyTerminalState() {
        return terminalState == TerminalState.GLOBALLY;
    }
this says in this case all recovery data is dropped. But if I set
execution.checkpointing.externalized-checkpoint-retention: RETAIN_ON_CANCELLATION
then I explicitely don't want my recovery data dropped? 😞
d
Ok, I wonder if this has to be higher than 1 for it to work even though that’s not documented?
Copy code
state.checkpoints.num-retained 1	Integer	The maximum number of completed checkpoints to retain.
Also have you tried with other versions of Flink besides 1.18?
c
I dont see maxNumberOfCheckpointsToRetain being considered anywhere in the codepath that leads to the deletion of the counter from the configmap
I only tried with flink 1.17.2, moving to 1.18 requires a lot of changes due to the removal of the scala API so we haven't tackled that yet
d
1.19?
Ok, so you have only tried on 1.17.2?
c
we had this issue in older versions as well, though we didn't manage to identify exactly what happened until now
d
Ok, I have not looked at the source but it does seem there might be a conflict between setting a max number of checkpoints to retain at 1 and doing a RETAIN_ON_CANCELLATION because perhaps that’s more than 1 to fulfill this?
And when does the pool eviction occur? It is predictable?