Hey Everyone, TLDR: Job Manager HA on Kubernetes Has anyone been able to make JobManager HA work with the Flink Kubernetes operator e2e? We are using flink 1.16 and flink k8s op 1.5.0. We followed this doc to set the properties but we faced 2 problems: 1. HTTP Requests by an operator to JM are load balanced by k8s service created by Flink operator between leader/slave in JM leading to 404(say jar was uploaded to slave JM but job submit request went to Master) 2. In case of Metaspace errors sometimes we see the JM deployment replicas are set to 1(i.e. only 1 JM pod is running after some time, initially 2 JM pods run), so we checked the flink deployments spec status.reconcilationSpec.lastReconciledSpec/lastStableSpec and could see spec.jobManager.replicas are set to 2. Any suggestion will be appreciated.

Hi, we are using 1.6.0 operator with 1.14 cluster and having issue when enabling HA as it is using hostname for HA vs IP address. Have you seen this issue? https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/ha/overview/

Hey, yes we faced exact same issue of operator using hostname(k8s service) which load balances between the two JM. For the fix, we patched the flink operator internally and fetched leader JM IP and used it to make requests

can you share the code snippet of this change? I was able to resolve this with only lastest 1.6.0 operator and 1.17 cluster. Do you know if there is bug Jira to fix this?

I have similar issue with Operator 1.6.0 and Flink 1.17, anyone who have fixed it?

@Hussain Abbas I don’t see this issue using 1.6.0 operator and flink 1.17 version