Has anyone been able to successfully submit jobs to the Flink session cluster with multiple Job Managers(HA enabled) using the official flink k8s operator? We are seeing that the JM running as leader/slave but the flink operator submits the job on the sessions cluster's rest K8s service which load balances requests between the JM(and not just to the leader) and fails sometimes with jar not found. Sorry for cross-posting. We enabled debug logs but couldn't figure out how to make the Flink operator submit job's to leader JM.

This sounds strange, I would expect only the leader to get requests

We enabled debug mode and could see that out of one of two JM, one was elected as Leader and the config map was updated with this JM pod's IP e.g. below.

org.apache.flink.k8s.leader.dispatcher: a42ea12d-469d-498c-9456-236ef2488c85,<akka.tcp://flink@192.168.56.249:6123/user/rpc/dispatcher_1>
  org.apache.flink.k8s.leader.ffffffffe56351a70000000000000001: e4e875f0-28a9-4866-b79e-990f9cef88d3,<akka.tcp://flink@192.168.46.152:6123/user/rpc/jobmanager_4>
  org.apache.flink.k8s.leader.resourcemanager: a42ea12d-469d-498c-9456-236ef2488c85,<akka.tcp://flink@192.168.56.249:6123/user/rpc/resourcemanager_0>
  org.apache.flink.k8s.leader.restserver: a42ea12d-469d-498c-9456-236ef2488c85,<http://192.168.56.249:8081>

Here is the jar not found exception in case that helps: https://ideone.com/qyoE2x Also we enabled debug logs for operator, and could see it's submitting the job to:

http://<flink-session-cluster-name>-rest.<k8s-namespace>/jars/<jard-id>/run

which is a type: ClusterIP service having selector labels to select both pods.