Hi all, we are running our clusters(1.19.1) on AW...
# troubleshooting
t
Hi all, we are running our clusters(1.19.1) on AWS EKS(Application mode) , the states are stored on S3. The clusters are managed by Flink Kubernetes operator (1.11.0). In addition, we are running Istio within the cluster, we had to exclude Pekko & JM RPC ports from being redirected to Envoy using the following annotation [1] [2]
Copy code
"traffic.sidecar.istio.io/excludeOutboundPorts"
everything was working fine and very stable, yet once Istio was updated from 1.22.x to 1.24.x. the communication between TM and JM terminated with following warning
[Warning] [{pekkoAddress=pekko.tcp://flink@10.233.24.57:6123, pekkoSource=pekko.tcp://flink@10.233.24.57:6123/system/endpointManager/reliableEndpointWriter-pekko.tcp%3A%2F%2Fflink%4010.233.90.141%3A6123-0, pekkoTimestamp=10:56:31.006UTC, pekkoUid=6633221187301835071, sourceActorSystem=flink, sourceThread=flink-pekko.remote.default-remote-dispatcher-6}] [o.a.p.r.ReliableDeliverySupervisor]: Association with remote system [pekko.tcp://flink@10.233.90.141:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [pekko.tcp://flink@10.233.90.141:6123]] Caused by: [The remote system explicitly disassociated (reason unknown).
Once we restarted the Flink deployment the following warnings appeared
[Warning] [{}] [o.a.f.r.r.a.ActiveResourceManager]: Discard registration from TaskExecutor dev-p2a2-taskmanager-1-1 at (pekko.tcp://flink@10.233.83.38:6122/user/rpc/taskmanager_0) because the framework did not recognize it
then
[Warning] [{}] [o.a.f.r.d.StandaloneDispatcher]: Ignoring JobGraph submission 'dev-p2a2' (5443fa441f7b39f55a48528d5665f17a) because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.
we deleted the state folder on S3 it did not work. the cluster config map which holds the last checkpoint(managed by the k8s-operator) was not there either. The only solution was to re-create the cluster (obviously we cannot do that in production ). I read there were many breaking changes between Istio 1.22.x and 1.24.x, yet after recreating the cluster it worked fine. any suggestions on why the cluster reached global terminate state and could not recover? [1] https://doc.akka.io/libraries/akka-management/current/bootstrap/istio.html#istio [2] https://pekko.apache.org/docs/pekko/1.0/project/migration-guides.html#migration-to-apache-pekko
πŸ‘ 1