Tamir Sagi
04/24/2025, 7:24 AM"traffic.sidecar.istio.io/excludeOutboundPorts"
everything was working fine and very stable, yet once Istio was updated from 1.22.x to 1.24.x. the communication between TM and JM terminated with following warning
[Warning] [{pekkoAddress=pekko.tcp://flink@10.233.24.57:6123, pekkoSource=pekko.tcp://flink@10.233.24.57:6123/system/endpointManager/reliableEndpointWriter-pekko.tcp%3A%2F%2Fflink%4010.233.90.141%3A6123-0, pekkoTimestamp=10:56:31.006UTC, pekkoUid=6633221187301835071, sourceActorSystem=flink, sourceThread=flink-pekko.remote.default-remote-dispatcher-6}] [o.a.p.r.ReliableDeliverySupervisor]: Association with remote system [pekko.tcp://flink@10.233.90.141:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [pekko.tcp://flink@10.233.90.141:6123]] Caused by: [The remote system explicitly disassociated (reason unknown).
Once we restarted the Flink deployment the following warnings appeared
[Warning] [{}] [o.a.f.r.r.a.ActiveResourceManager]: Discard registration from TaskExecutor dev-p2a2-taskmanager-1-1 at (pekko.tcp://flink@10.233.83.38:6122/user/rpc/taskmanager_0) because the framework did not recognize it
then
[Warning] [{}] [o.a.f.r.d.StandaloneDispatcher]: Ignoring JobGraph submission 'dev-p2a2' (5443fa441f7b39f55a48528d5665f17a) because the job already reached a globally-terminal state (i.e. FAILED, CANCELED, FINISHED) in a previous execution.
we deleted the state folder on S3 it did not work. the cluster config map which holds the last checkpoint(managed by the k8s-operator) was not there either.
The only solution was to re-create the cluster (obviously we cannot do that in production ).
I read there were many breaking changes between Istio 1.22.x and 1.24.x, yet after recreating the cluster it worked fine.
any suggestions on why the cluster reached global terminate state and could not recover?
[1] https://doc.akka.io/libraries/akka-management/current/bootstrap/istio.html#istio
[2] https://pekko.apache.org/docs/pekko/1.0/project/migration-guides.html#migration-to-apache-pekko