Jérôme Gagnon
10/08/2024, 5:42 PMJérôme Gagnon
10/08/2024, 5:54 PMLiad Shachoach
10/08/2024, 8:13 PMJérôme Gagnon
10/08/2024, 9:07 PMreconcilationStatus:
state: DEPLOYEDJérôme Gagnon
10/08/2024, 9:10 PMJérôme Gagnon
10/08/2024, 9:10 PMGyula Fóra
10/09/2024, 2:28 PMJérôme Gagnon
10/09/2024, 6:21 PMJérôme Gagnon
10/09/2024, 6:23 PMLiad Shachoach
10/09/2024, 7:24 PMJérôme Gagnon
10/09/2024, 7:45 PMGyula Fóra
10/09/2024, 7:48 PMGyula Fóra
10/09/2024, 7:48 PMGyula Fóra
10/09/2024, 7:48 PMGyula Fóra
10/09/2024, 7:49 PMJérôme Gagnon
10/09/2024, 7:50 PMGyula Fóra
10/09/2024, 7:51 PMJérôme Gagnon
10/09/2024, 7:53 PMGyula Fóra
10/09/2024, 7:55 PMGyula Fóra
10/09/2024, 7:55 PMJérôme Gagnon
10/09/2024, 8:01 PMJérôme Gagnon
10/16/2024, 6:17 PMLiad Shachoach
10/16/2024, 6:20 PMJérôme Gagnon
10/16/2024, 6:24 PMLiad Shachoach
10/16/2024, 6:26 PMJérôme Gagnon
10/16/2024, 6:29 PMYou can always disable savepoints on the flinkdep, suspend and restore from savepoint.That’s what we’ve been doing (as per my original message) but it doesn’t feel right ? I could probably try using this method however : https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/job-management/#redeploy-using-the[…]epointredeploynonce
Liad Shachoach
10/16/2024, 6:31 PMJérôme Gagnon
10/16/2024, 6:33 PMLiad Shachoach
10/16/2024, 6:35 PMJérôme Gagnon
10/16/2024, 9:23 PMsavepointRedeployNonce and the initialSavepointPath the operator doesn’t re-create the deployment for the job that had failed savepoint on cancel. Which lines up with what I’ve previously observed.Jérôme Gagnon
10/16/2024, 9:29 PM2024-10-16 21:10:06,085 WARN org.apache.flink.kubernetes.operator.service.AbstractFlinkService [] - Error while Scaling JobManager Deployment to zero - io.fabric8.kubernetes.client.KubernetesClientTimeoutException: Timed out waiting for [60000] milliseconds for [Pod] with name:[null] in namespace [REDACTED].Liad Shachoach
10/17/2024, 5:37 AMJérôme Gagnon
10/29/2024, 3:32 PMsavepointRedeployNonce to flinkdep
4. … nothing happens - operator logs state nothing has to be reconciled
5. flink-operator pods restart
6. The job upgrades with a forced redeployment du to savepointRedeployNonce
cc; @Liad Shachoach @Gyula Fóra do you think that’s plausible ? What could be causing that, and how to mitigate it ?Gyula Fóra
10/29/2024, 3:34 PMJérôme Gagnon
10/29/2024, 3:35 PM