👋 How does the Flink Kubernetes Operator determine if a job is healthy? From the docs:

* When HA is enabled the

savepoint

upgrade mode may fall back to the

last-state

behaviour in cases where the job is in an unhealthy state.

I'm wondering about the unhealthy state because I'm observing an issue where the operator won't take a new savepoint when

savepointTriggerNonce

is updated, or when a FlinkDeployment spec is changed and the

upgradeMode

is set to

savepoint

. Any help is appreciated!

It would be good to first understand why the job doesn't go into running state . Is there anything in the log to hint at the problem? Connectivity issue or something?

I'm getting the same result (job stuck in RECONCILING) for the basic-example too. When I upgrade the spec with

upgradeMode: savepoint

and

kubernetes.operator.job.upgrade.last-state-fallback.enabled: false

there are these logs in the operator:

2023-02-28 18:46:42,779 o.a.f.k.o.r.d.AbstractFlinkResourceReconciler [INFO ][redacted] UPGRADE change(s) detected (FlinkDeploymentSpec[job.upgradeMode=SAVEPOINT,flinkConfiguration.web.timeout=1800000] differs from FlinkDeploymentSpec[job.upgradeMode=STATELESS,flinkConfiguration.web.timeout=1800001]), starting reconciliation.
2023-02-28 18:46:42,793 o.a.f.k.o.l.AuditUtils         [INFO ][redacted] >>> Event  | Info    | SPECCHANGED     | UPGRADE change(s) detected (FlinkDeploymentSpec[job.upgradeMode=SAVEPOINT,flinkConfiguration.web.timeout=1800000] differs from FlinkDeploymentSpec[job.upgradeMode=STATELESS,flinkConfiguration.web.timeout=1800001]), starting reconciliation.
2023-02-28 18:46:42,793 o.a.f.k.o.r.d.AbstractJobReconciler [INFO ][redacted] Upgrading/Restarting running job, suspending first...
2023-02-28 18:46:42,794 o.a.f.k.o.r.d.ApplicationReconciler [INFO ][redacted] Job is not running and HA metadata is not available or usable for executing the upgrade, waiting for upgradeable state

How does the operator make the FlinkDeployment transition to Running state?

really seems like there is some connectivity issue between the operator and the submitted flink jobs, is that possible?

Yeah that is definitely what it seems like, it's possible. I'm looking into it. When I follow the Quick Start using minikube, this issue isn't present. There may be some kind of networkpolicy or other that is deployed in the environment where the issue exists. Our flink operator is deployed such that it manages all namespaces (cluster-scoped), any ideas on potential sources of connectivity issues?

you probably need to use the podTemplate feature to add the required labales etc. This is very common in prod envs