Hi Team 👋 , I’ve few questions around HA while using flink operator 1.4.0. On my FlinkDeployment, upgradeMode is stateless. This is because I don’t have a distributed storage option available yet for checkpoint/savepoint. So for the same reason I don’t have jobmanager HA configured. From the doc, it’s a requirement to provide high-availability.storageDir for Jobmanager HA. Can HA work without a distributed storage? Now I’ve deployed the FlinkDeployment streaming job with single jobmanager and two taskmanagers. The job source from kafka topic , do processing in flink , sink to another kafka topic. To test the reliability of the taskmanagers, I killed one of the taskmanager and the whole 1 jobmanager + 2 taskmanagers were shutdown after a new one pod been tried to spin up, resulting in Application Status: FAILED with “_Caused by: org.apache.flink.runtime.JobException: Recovery is suppressed by NoRestartBackoffTimeStrategy”_. Any pointers of what might be the cause? Ideally I’m looking for JM and TM to be highly available without a distributed storage while deployed jobs are stateless. Thanks for your time!

You are confusing a few things here. Just because your deployment upgradeMode is stateless doesn’t mean that your job is stateless. Kafka sources are stateful. The error you get is because jobs without checkpointing enabled have no default restart strategy. If any failure happens the job fails with this error. You can either enable checkpointing to get a consistent recovery behaviour or simply add a restart strategy without checkpointing but that may lead to data loss at the kafka consumers (because they are stateful). You will not be able to get JobManager fault tolerance without a distributed storage. So if the JM fails the job fails. In case of Kubernetes it may actually just restart the job similar to a TM failure but it will not know what checkpoint to use.

@Gyula Fóra sorry for the confusion and clarifying it. I’ve ‘latest’ offset setting for kafka topics in the job and several of those jobs don’t require the state to be maintained. For those, with your help, now I added restart strategy of ‘fixed-delay’ and killing any TMs or JM recovers perfectly. That’s exactly what I need for my stateless jobs use-case to make sure it’s resilient to failures at infra level. Thank you so much for your timely help! BTW, I watched your interview with @rmetzger along with @Matyas Orhidi . Great questions and you guys clarified all in pretty detail. I searched docs for several clarifications and this video gave me further greater insights. Thanks again for all the great work!