With Flink operator (1.5) how do we get out of the...
# troubleshootingd
Dylan Meissner
09/07/2023, 2:58 PMWith Flink operator (1.5) how do we get out of the RECOVERDEPLOYMENT/RESTOREFAILED loop when the JobManager deployment is accidentally deleted?
Dylan Meissner
09/07/2023, 2:59 PMWe can no longer delete the flinkdeployment (which is what should have been done to cancel the job).
Dylan Meissner
09/07/2023, 4:24 PMIt’s hard to repro, but somehow it is in this state. It won’t take any new jobs while it’s in this loop.
g
Gyula Fóra
09/07/2023, 7:10 PMIt’s strange that you cannot delete the flinkdeployment
Gyula Fóra
09/07/2023, 7:10 PMcan you please open a jira ticket with the logs?
d
Dylan Meissner
09/07/2023, 7:15 PMWe worked through it by updating the operator config
kubernetes.operator.job.upgrade.last-state-fallback.enabledfalseDylan Meissner
09/12/2023, 4:22 PMAs a follow-up, the deployments were running to completion like a batch-style job. When the next deployment ran, it hit FLINK-32111 and became un-deletable as it was always busy reconciling.
We are looking to update to Operator version 1.6 to pick up the fix, and also have advised the engineers to leverage
kubernetes.operator.jm-deployment.shutdown-ttlDylan Meissner
09/12/2023, 4:32 PMDocs are sparse on best approach to running Flink deployments this way. If I threw something into an MR, it would probably be very naive but we could iterate with feedback. What do you think?
g
Gyula Fóra
09/13/2023, 5:57 AMinto a PR you mean?
d
Dylan Meissner
09/13/2023, 12:40 PMYes, I mean a PR.
g
Gyula Fóra
09/13/2023, 1:04 PMsounds good 🙂
👍 1
10 Views