Hi Folks Wondering if anyone had have issue where ...
# troubleshooting
j
Hi Folks Wondering if anyone had have issue where the flink k8s operator doesn’t replace the Deployment image even if the FlinkDeployment one has been changed ? Flink 1.19 Operator 1.9
In fact, nothing gets reconciled but the FlinkDeployment shows no events.
l
What is the status of the flinkdep? I had some cases where the job manager was unable to take a savepoint and it prevented reconciliation
j
Copy code
reconcilationStatus:
  state: DEPLOYED
Super strange to, we tried to manually delete the flink deployment and the finalizer weren’t getting cleared, we removed the finalizer manually, then recreated the flinkdeployment (with a savepoint) and only then the operator started to try to reconcile it; but was in bad state (jobmanager has been killed or something) then we went through another round of manual delete, this time the finalizers got cleared and then the recreation/recovery got reconciled and deployment created.
It almost looks like the operator lost track of the flinkdep at some point and stop trying to reconcile it ?
g
Could it be some issue with the operator itself? Maybe connectivity or lost access to the resources etc.
j
I can’t find anything obvious in the logs, just the “Resource fully reconciled, nothing to do” - Im assuming if this was an RBAC/connectivity issue there would be error logs right ?
At this point I’m pretty sure it has to do with the operator but not sure what yet.. almost like it’s a bug/internal logic issue
l
From my experience with the operator so far, it is more likely related to an issue with the job manager. Probably related to it not being able to take a savepoint. If this happens again, could you share the output of “kubectl get flinkdep”?
j
I will do that for sure
g
I agree probably not an operator issue ;) my guess would be Kubernetes infra / rbac problem
If the operator doesn’t reconcile the resource
The question is , when you change the resource spec does the operator react
If not then something is off
j
It does for all our flinkdep (50+), except when it doesn’t like in that specific case.
g
Seen similar issues in the past but those have been fixed in fabric8 a long time ago
j
We also did have that issue on savepoint timeout previously since this job state is fairly big but we were seeing logs in the operator for such problems.
g
Did you try debug logs?
On the operator
j
yes, next time this happen ill have more infos
Hey folks; so it happened again, but this time I see that the Savepoint before restart has failed (@Liad Shachoach hunch was correct) for one job that doesn’t get updated properly. That’s most likely the cause; however I remember that previously the operator was reporting a failure in CRD events and it wouldn’t get stuck like that.
l
I think this is not even related to the operator. Once a savepoint fails it will not be able to suspend or update the flinkdep. You should probably focus on why the savepoint process failed 🙂 As far as i remember, this happened to me in previous operator versions as well
j
The savepoint failing is definitely not the operator fault. The fact that it stops syncing the Deployment to the FlinkDep however is an issue because I can’t even do a manual restore with checkpoint for instance.
l
You can always disable savepoints on the flinkdep, suspend and restore from savepoint. I am not sure if that’s the right flow but it should work. When this happens to us we usually delete the flinkdep and deploy it again (configuring it to start from a savepoint / checkpoint)
j
You can always disable savepoints on the flinkdep, suspend and restore from savepoint.
That’s what we’ve been doing (as per my original message) but it doesn’t feel right ? I could probably try using this method however : https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#redeploy-using-the[…]epointredeploynonce
l
This is what we do 🙂
j
I still think the operator should at least emit a k8s event for the Flinkdep to indicate the failure (and Im pretty sure it did previously)
l
Yea that’s true
j
I might be missing something but even after adding
savepointRedeployNonce
and the
initialSavepointPath
the operator doesn’t re-create the deployment for the job that had failed savepoint on cancel. Which lines up with what I’ve previously observed.
Copy code
2024-10-16 21:10:06,085 WARN  org.apache.flink.kubernetes.operator.service.AbstractFlinkService [] - Error while Scaling JobManager Deployment to zero - io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [60000] milliseconds for [Pod] with name:[null] in namespace [REDACTED].
l
You will need to delete the flinkdep object and recreate it with the savepoint values. In the documentation it is stated that you can “redeploy” the flinkdep with these configs
j
Circling back here after a while… this happened once again. I’m pretty sure i found the conditions for it when it hapenned. When a job fails to upgrade due to a failed savepoint-on-cancel then the operator simply stop trying to upgrade the job altogether. Restarting the operator pods makes it track the job correctly again. The last occurence history 1. Job upgrade trigger 2. Savepoint-on-cancel fails 3. Adding
savepointRedeployNonce
to flinkdep 4. … nothing happens - operator logs state nothing has to be reconciled 5. flink-operator pods restart 6. The job upgrades with a forced redeployment du to
savepointRedeployNonce
cc; @Liad Shachoach @Gyula Fóra do you think that’s plausible ? What could be causing that, and how to mitigate it ?
g
The upgrade flow has been reworked to some extent in the new 1.10 release. Could you maybe try it with that?
j
I can most definitely try that, missed the release!